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SUMMARY 


Objective 

The  objective  of  this  study  was  to  design  and  conduct  a  Training 
Capabilities  Test  (TCT)  of  the  2D  subsystem  of  the  Electronic  Equipment 
Maintenance  Trainer  (EEMT)  within  the  Primary  Power  portion  of  the  AN/SPS-10 
radar  training  course  at  ET  "A"  School,  Great  Lakes  Naval  Training  Center, 
Illinois.  The  TCT  was  designed  to  determine  the  training  effectiveness, 
operational  suitability,  and  cost  of  the  2D  EEMT  as  it  supports  the  training  system 
in  a  specific  training  environment. 


Background 

Operational  readiness  of  Naval  electronic  equipment  depends,  in  large 
part,  on  the  ability  of  enlisted  technicians  to  perform  appropriate  maintenance 
actions.  Effective  technician  training,  which  must  be  accomplished  at  reasonable 
cost,  is  then  a  necessary  condition  for  military  preparedness.  Based  on  the 
assumption  that  maximum  transfer  of  training  will  occur  if  technicians  are 
trained  on  the  same  equipment  that  they  are  subsequently  required  to  maintain, 
the  Navy  has  traditionally  used  operational  equipment  to  assist  instructors  in  a 
variety  of  training  tasks.  The  use  of  operational  equipment,  however,  can  be 
costly,  dangerous,  inflexible,  unsupported  by  training  materials,  and  in  some 
instances  the  fleet  may  be  deprived  of  their  full  allotment  of  operational 
hardware.  With  the  rapid  changes  in  technology  and  the  increased  complexity  and 
cost  of  electronic  devices,  more  often  than  not,  schools  are  forced  to  train 
students  on  outdated  equipment  which  is  itself  difficult  to  maintain,  particularly 
with  the  limited  availability  of  parts  and  the  low  priority  assigned  to  training 
needs  in  the  supply  system.  The  2D  EEMT  was  designed  and  developed  to 
overcome  many  of  the  problems  associated  with  using  operational  equipment  for 
training  electronics  technicians. 


Approach 


The  TCT  was  composed  of  a  series  of  studies  designed  to  evaluate  the 
training  effectiveness,  operational  suitability  and  cost  effectiveness  of  the  2D 
EEMT  system  for  training  Navy  electronics  technicians.  Three  training  modes 
were  selected  for  study:  an  operational  equipment  only  mode,  a  low  EEMT  mode 
in  which  the  2D  EEMT  was  used  for  approximately  30  percent  of  training,  and  a 
high  2D  EEMT  mode  in  which  the  2D  EEMT  was  used  for  approximately  60  percent 
of  training.  In  addition,  two  training  formats,  individual  and  pair,  were  employed. 
The  total  amount  (in  hours)  of  laboratory  training  was  the  same  in  all  conditions. 

Existing  measures  and  tracking  procedures  were  incorporated  to  the 
greatest  extent  possible.  Scores  from  the  ET  "A"  School  Performance  Test  (PT) 
were  the  principal  measure  of  2D  EEMT  training  effectiveness.  Student  and 
instructor  attitudinal  data  pertaining  to  the  use  of  the  2D  EEMT  and  operational 
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equipment  in  training  were  obtained.  Limited  follow-up  data  were  collected  to 
track  the  need  for  retraining  subsequent  to  "A"  School.  Operational  suitability 
data  were  collected  tc  determine  2D  EEMT  reliability,  maintainability, 
availability  for  training,  appropriateness  for  Primary  Power  training,  and 
adaptability  to  other  training  areas.  The  respective  costs  of  the  2D  EEMT  and 
operational  equipment  over  a  15-year  life  cycle  were  compared.  The  comparative 
costs  of  the  2D  EEMT  and  operation  equipment  in  selected  training  scenarios  were 
also  examined. 


Results 

The  accuracy  of  troubleshooting  performance  was  essentially  equal  when 
students  were  trained  on  either  the  2D  EEMT  or  operational  equipment.  Students 
trained  on  operational  equipment  appeared  more  efficient  than  those  who 
received  2D  EEMT  training,  and  students  trained  as  individuals  were  slightly  more 
efficient  than  those  trained  in  pairs.  The  18  2D  EEMTs  were  available  for  about 
72  percent  of  the  time  for  both  laboratory  and  extra  study/remediation  use.  The 
life  cycle  cost  comparison  favored  the  2D  EEMT  over  the  operational  equipment 
if  at  least  three  units  are  acquired.  Considering  investment  costs  only,  a  50 
percent  savings  was  estimated  for  the  purchase  of  20  2D  EEMT  units  as  opposed 
to  20  AN/SPS-10  units.  Instructors  and  students  were  in  agreement  that  the  2D 
EEMT  had  an  important  and  useful,  but  limited,  role  as  a  supplement  to 
operational  equipment  training. 


Conclusions  and  Recommendations 

When  used  in  conjunction  with  operational  equipment,  the  2D  EEMT  can 
reduce  reliance  on  operational  equipment  trainers  without  a  reduction  in  training 
effectiveness.  The  2D  EEMT  is  equally  effective  for  low,  middle,  and  high  ability 
students  and  is  an  effective  alternative  to  operational  equipment  for  extra  study 
and  remediation.  It  provides  increased  opportunity  for  troubleshooting  practice 
due  to  reduced  safety  and  instructor  monitoring  requirements. 

The  2D  EEMT  is  easily  transportable,  compatible  with  the  ET  "A"  School 
environment,  and  adaptable  to  other  training  environments.  Maintainability  and 
reliability  of  the  2D  EEMT,  however,  would  be  enhanced  by  provision  of  a 
comprehensive  spare  parts  package  and  modifications  to  the  touch  panels  used  for 
student  input. 

The  life  cycle  cost  savings  realized  by  using  2D  EEMTs  instead  of 
operational  equipment  is  largely  dependent  on  the  initial  investment  costs 
associated  with  the  actual  system  in  question. 
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I.  INTRODUCTION 


This  report  presents  the  results  of  the  Training  Capabilities  Test  (TCT) 
of  the  Electronic  Equipment  Maintenance  Trainer  (EEMT).  This  test  and 
evaluation  was  conducted  during  Phase  II  of  contract  no.  N61 339-81 -C-0 126  in 
accordance  with  the  approved  TCT  plan  (Cicchinelli,  Keller,  3c  Harmon,  1982) 
developed  during  Phase  I  of  this  contract.  The  plan  was  designed  to  respond  to 
the  evaluation  requirements  of  the  Naval  Decision  Coordinating  Paper  (NDCP) 
(Department  of  the  Navy,  1978)  and  the  Device  Test  and  Evaluation  Master  Plan 
(DTEMP)  (Pine,  Daniels,  3c  Malec,  1981)  for  the  EEMT  and  subsequently  modified 
to  correspond  with  the  use  made  of  the  EEMT  by  the  Electronic  Technician  (ET) 
"A"  School,  Service  School  Command,  Great  Lakes,  Illinois. 


Problem 


Operational  readiness  of  Naval  electronic  equipment  depends,  in  large 
part,  on  the  ability  of  enlisted  technicians  to  perform  appropriate  maintenance 
actions.  Effective  technician  training,  which  must  be  accomplished  at  reasonable 
cost,  is  then  a  necessary  condition  for  military  preparedness.  Based  on  the 
assumption  that  maximum  transfer  of  training  will  occur  if  technicians  are 
trained  on  the  same  equipment  that  they  are  subsequently  required  to  maintain, 
the  Navy  has  traditionally  used  operational  equipment  to  assist  instructors  in  a 
variety  of  training  tasks.  In  recent  years,  experience  has  dictated  that  the  use  of 
operational  equipment  in  training  environments  is  not  always  the  optimal 
approach  to  instruction. 

The  use  of  operational  equipment  used  in  a  training  capacity  can  be 
costly,  dangerous,  inflexible,  unsupported  by  training  materials,  and  in  some 
instances  the  fleet  may  be  deprived  of  their  full  allotment  of  operational 
hardware.  With  the  rapid  changes  in  technology  and  the  increased  complexity  and 
cost  of  electronic  devices,  more  often  than  not,  schools  are  forced  to  train 
students  on  outdated  equipment  which  is  itself  difficult  to  maintain,  particularly 
with  the  limited  availability  of  parts  and  the  low  priority  assigned  to  training 
needs  in  the  supply  system. 

The  EEMT  was  designed  and  developed  as  a  generic  trainer  intended  to 
overcome  many  of  the  problems  associated  with  using  operational  equipment  for 
training  electronics  technicians. 


Purpose 


The  purpose  of  this  investigation  was  to  design  and  conduct  a  TCT  of  the 
EEMT.  Testing  of  training  devices  in  an  operational  environment  is  more  limited 
in  scope  than  the  Operational  Test  and  Evaluation  required  for  operational 
hardware.  The  TCT  was  designed  to  determine  the  training  effectiveness, 
operational  suitability  and  cost  of  the  EEMT  as  it  supports  the  training  system  in 
a  specific  training  environment. 


Practice  of  operation  and  maintenance  tasks  is  a  critical  element  of  "A" 
School  electronics  training  in  the  Navy.  Troubleshooting  and  hands-on  training 
have  been  accomplished  primarily  through  the  use  of  operational  equipment.  The 
Operational  Requirement,  PN-50  (1976),  outlined  a  number  of  problems  with  this 
approach  and  established  general  goals  and  criteria  for  the  training  system.  In 
response,  the  Navy  initiated  an  engineering  development  effort  to  develop  a 
general  purpose  EEMT  system  that  could  reduce  reliance  on  operational 
equipment  at  the  "A"  School  level. 

The  EEMT  project  was  divided  into  four  phases:  (1)  concept  formulation, 
(2)  system  definition,  (3)  prototype  development,  and  (*f)  system  test  and 
evaluation.  A  basic  system  design  was  drafted  as  part  of  the  first  phase  of  the 
project.  At  the  same  time,  the  ET  School  at  Great  Lakes,  Illinois  and  the 
Electronics  Warfare  Technician  (EW)  School  at  the  Naval  Technical  Training 
Center,  Pensacola,  Florida  were  designated  as  the  target  users  of  the  EEMT 
(Wylie  &  Bailey,  1978).  Phase  II  resulted  in  the  prototype  design  selection  (Pine, 
Koch,  &  Malec,  1981).  Specifications  were  prepared  for  a  two-dimensional  (2D) 
subsystem  with  interactive  displays  and  a  three-dimensional  (3D)  subsystem  with 
simulated  generic  electronic  equipment.  The  prototype  development  phase  (Phase 
III)  resulted  in  two  subsystem  configurations.  The  final  project  phase,  system  test 
and  evaluation,  is  the  focus  of  this  report. 

The  EEMT  system,  and  the  2D  subsystem  in  particular,  js  the  prototype 
version  of  the  Generalized  Maintenance  Training  Simulator  (GMTS)  also  developed 
by  the  Navy  Personnel  Research  and  Development  Center  (Towne  &  Munro,  198J). 
The  purpose  of  the  GMTS  was  to  demonstrate  the  feasibility  of  the  basic  system 
concepts  and  technology  and  to  facilitate  the  development  of  prototype 
specifications.  While  the  hardware  has  changed  substantially  during  this  evolution 
from  GMTS  to  EEMT,  the  functional  characteristics  of  the  two  training  systems 
remain  essentially  the  same. 

To  date,  the  GMTS  has  undergone  three  field  evaluations  in  different  ET 
training  settings  (Rigney,  Towne,  King,  &  Moran,  1978;  Rigney,  Towne,  Moran,  & 
Mishler,  1978;  Towne,  Munro,  <5c  Johnson,  1983).  Based  on  the  favorable  results  of 
these  studies,  the  training  effectiveness  of  a  2D  trainer  has  already  been 
demonstrated,  at  least  in  principle,  and  thus,  the  major  focus  of  this  TCT  should 
be  on  the  integrated  2D-3D  system  and  the  3D  EEMT  subsystem.  However,  the 
primary  focus  of  two  of  these  field  tests  was  to  demonstrate  the  feasibility  and 
applicability  of  the  GMTS  concept.  Neither  study  was  designed  to  evaluate  the 
training  effectiveness,  cost,  or  suitability  of  the  system  as  an  integral  part  of 
instruction.  Only  Towne,  Munro,  and  Johnson  (1983)  examined  the  training 
effectiveness  of  the  GMTS  in  a  training  laboratory  setting.  Thus,  the  TCT  of  the 
EEMT  system  was  originally  designed  to  give  equal  emphasis  to  the  training 
effectiveness,  operational  suitability  and  cost  of  both  the  2D  and  3D  subsystems, 
an  approach  directed  by  the  DTEMP  (Pine,  Daniels,  <5c  Malec,  1981). 


Modifications  to  the  TCT  Sco 


At  the  direction  of  the  Navy,  on  January  25,  1982,  the  EW  School  was 
officially  dropped  as  a  site  for  the  TCT.  On  February  16,  1983,  a  second  major 
change  in  the  TCT  was  made.  Since  the  3D  EEMT  subsystem  delivered  to  the  ET 
School  could  not  be  integrated  into  the  existing  curriculum  without  modifications 
to  the  3D  lessonware  delivered  and  substantial  additional  lessonware  development 
to  support  a  2-week  instructional  program,  the  evaluation  of  this  unit  was  deleted 
from  the  TCT  plan  (Cicchinelli,  Keller,  Sc  Harmon,  1982). 


System  Description 


The  functional  description  of  the  2D  EEMT  was  based  on  the  premise 
that  the  training  system  would  be  generic.  It  was  hypothesized  that  more 
students  could  be  trained  in  less  time,  more  cost  effectively,  and  to  a  greater 
proficiency  level  utilizing  the  EEMT  system  as  an  enhancement  to  current 
operational  equipment.  The  2D  EEMT  was  not  considered  as  a  replacement 
system  because  technical  training  requires  technical  equipment. 

The  2D  EEMT  system  was  intended  to  provide  trainees  with  practice  on 
generic  preventive  and  corrective  maintenance  procedures.  The  system  provided 
self-paced  instruction  adaptable  to  the  skills  and  needs  of  individual  trainees. 
Immediate  feedback  was  provided  on  trainee  performance  and  the  system 
provided  computer-assisted  and  limited  computer-managed  instruction  such  that 
student  performance  skills  were  tested  and  recorded. 

The  2D  hardware  consists  of  a  microcomputer,  image  and  adaptive 
displays  fitted  with  touch-sensitive  panels  for  user  input,  and  a  videodisc  player. 
A  portable  terminal/printer,  which  can  be  shared  among  2D  EEMT  consoles, 
provides  for  instructor  input  to  the  system,  lessonware  authoring,  and  access  to 
student  performance  data.  Figure  1  shows  the  2D  hardware  configuration, 
including  the  instructor  terminal  and  student  console  used  for  training.  The 
adaptive  display  presents  text,  problems,  and  menu  choices;  the  image  display 
presents  equipment  images  from  a  photographic  data  base.  This  data  base  is 
arranged  hierarchically  and  includes  an  overall  system  image  together  with 
successively  more  detailed  images  of  system  components  and  subcomponents. 

The  2D  unit  is  primarily  a  stand-alone,  equipment-specific  trainer  which 
uses  both  structured  and  free-play  lessonware.  Structured  lessons  provide 
instruction  on  various  topics  in  a  typical  interactive  tutorial  format.  Free-play 
lessons  present  troubleshooting  problems  and  students  are  allowed  to  observe  and 
manipulate  the  simulated  equipment  until  a  solution  is  reached.  Free-play  lessons 
closely  parallel  laboratory  training  problems. 
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Illustration  of  2D  F.EMT  student  station  with  the  instructor  terminal 


II.  APPROACH 


The  test  and  evaluation  of  the  2D  EEMT  was  conducted  in  accordance 
with  the  statement  of  work  (Revision  B,  2-16-83)  and  the  approved  TCT  plan 
developed  during  Phase  I  of  this  contract.  The  plan  specifically  addressed  three 
aspects  of  the  training  device:  training  effectiveness,  operational  suitability,  and 
cost.  To  expedite  the  collection  of  relevant  data,  the  plan  was  developed  to 
integrate  with  the  existing  training  environments  and  the  intended  uses  of  the  2D 
EEMT  in  the  training  system. 


The  Training  Environment 


The  test  and  evaluation  of  the  2D  EEMT  was  conducted  primarily  on-site 
in  the  ET  "A"  School  at  Great  Lakes  Naval  Training  Center. 

All  prospective  ET  students  receive  training  in  Basic  Electronics  and 
Electricity  (BE&E)  prior  to  reporting  to  the  ET  School  as  depicted  in  Figure  2.  ET 
students  are  then  given  Advanced  Electronics  training  followed  by 
Communications  instruction,  or  the  Fundamentals  of  Tubes  and  Transistors  for  ET 
students  in  the  nuclear  field.  The  length  of  training,  therefore,  depends  on 
whether  a  student  is  designated  ET  or  a  nuclear  ET.  Nuclear  trained  students 
comprise  about  one-third  of  each  ET  class  of  about  66  students,  and  an  effort  is 
made  to  consolidate  them  into  a  single  training  shift.  Approximately  22  students 
are  assigned  to  each  of  three  shifts  beginning  each  week.  As  Figure  2  shows,  all 
ET  students  are  then  merged  again  for  6  weeks  of  AN/SPS-10  Radar  training.  The 
primary  application  of  the  EEMT  system  was  in  this  final  course.  Specifically, 
eight  2D  EEMT  units  were  placed  in  the  Primary  Power  laboratory  section  (the 
first  2  weeks)  of  the  AN/SPS-10  Radar  Systems  instruction.  This  laboratory  had 
10  work  stations  using  operational  AN/SPS-10  equipment.  Radar  training  at  ET 
"A"  School  is  equipment  specific  and  uses  the  surface  search  AN/SPS-10 
equipment  for  all  hands-on  training  and  testing.  Additional  2D  EEMT  units  were 
placed  in  the  extra  study/remediation  room  associated  with  Radar  Systems. 

Existing  student  assessment  methods  and  record  keeping  procedures  were 
identified  as  part  of  the  course  review.  Performance  Tests  (PTs),  Unit  Tests, 
maintenance  forms,  and  other  data  collection  instruments  were  also  identified. 
PTs  for  the  radar  segment  of  instruction  are  indicated  in  Figure  2. 

During  the  planning  phase,  ET  School  personnel  made  two  points  very 
clear:  (1)  the  EEMT  would  play  only  a  supplemental  role  in  instruction  and  (2)  the 
final  decision  regarding  acceptance  of  the  units  (hardware  and  lessonware)  would 
be  dependent  on  the  findings  of  an  on-site  shakedown  period.  The  use  of  the 
AN/SPS-10  as  a  representative  radar  trainer  fills  a  greater  need  than  simply 
generic  training,  because  nearly  every  ship  has  an  AN/SPS-10,  and  there  is  no  "C" 
School  for  this  radar  system.  For  this  reason,  ET  personnel  stated  that  it  could 
not  be  effectively  replaced  as  a  training  tool  by  the  generic  EEMT  system. 
Furthermore,  instructors  reported  that  students  are  "too  sophisticated"  for 
generic  trainers  by  the  time  they  reach  ET  School.  Specific  issues  identified 
during  preliminary  on-site  interviews  included: 
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Figure  2.  Illustration  of  the  student  flow  in  the  ET  School. 


*ET(NT)~electronics  technicians  trained  in  the  nuclear  field  specialty  area. 
tPT  is  the  performance  test  for  a  given  training  segment. 


•  The  main  use  of  the  2D  EEMT  subsystem  was  to  be  during 
the  first  2- week  area  of  Radar  Systems  (Primary  Power). 
The  first  week  focused  on  general  radar  issues,  while  the 
second  week  focused  on  the  AN/SPS- 10  power  supply. 

•  The  2D  units  were  to  be  dispersed  among  existing 
laboratories  and  remedial  work  areas.  In  the  laboratories, 
only  free-play  problems  were  to  be  used.  Students  were  to 


be  rotated  between  the  2D  EEMTs  and  the  operational 
equipment  stations. 

•  Additional  lesson  ware  was  to  be  developed  by  the 
Instructional  Program  Development  (IPD)  Center  for  the 
receiver/transmitter  portions  of  AN/SPS- 10  training  and 
for  the  Communications  Systems  block  of  instruction  that 
precedes  radar. 

•  The  EEMT  was  to  be  considered  particularly  well  suited  for 
extra  study/remedial  training  because  of  its  accessibility 
and  the  fact  that  virtually  no  instructor  monitoring  was 
required.  Structured  lessons  were  considered  to  be  suitable 
only  for  remedial  instruction. 


The  Training  Capabilities  Test  Plan 


The  test  and  evaluation  plan,  developed  during  Phase  I  of  this  contract, 
outlined  a  series  of  studies  designed  to  evaluate  the  training  effectiveness, 
operational  suitability,  and  cost  effectiveness  of  the  EEMT  system  for  training 
Navy  electronics  technicians.  This  modular  approach  was  selected  for  the 
following  reasons: 

•  The  potential  impacts  of  unscheduled  changes  in  the 
training/testing  curriculum  subsequent  to  TCT  start-up 
would  be  minimized  because  they  could  be  restricted  to  a 
particular  evaluation  element.  Thus,  the  entire  test  and 
evaluation  would  not  be  compromised  by  deviations  from 
current  training  practices. 

•  This  approach  provided  the  flexibility  necessary  to  be 
responsive  to  changing  environmental  constraints. 

Variability  in  factors  such  as  instructor  availability, 
student  flow,  and  equipment  reliability  would  have  little 
impact  on  evaluation  outcomes. 

•  The  focus  of  each  evaluation  element  could  be  directed 
toward  a  particular  instructional  area  without  affecting 
other  components  of  the  plan.  In  this  way,  each  element 
was  focused  on  the  most  relevant  training  circumstances 
available. 


•  This  approach  allowed  better  coordination  between 
evaluation  efforts  and  the  projected  EEMT  implementation 
schedule;  data  collection  could  proceed  for  one  study  while 
the  planning  and  implementation  for  a  second  study  was 
still  in  progress. 

The  TCT  placed  only  minimal  demands  on  the  ET  School  environment. 
Existing  measures  and  tracking  procedures  were  incorporated  to  the  greatest 
extent  possible.  School  personnel  had  only  a  few  additional  data  collection 
responsibilities  as  a  consequence  of  the  evaluation  effort.  Thus,  administrative 
and  technical  assistance  were  the  primary  types  of  support  required  from  ET 
School  personnel. 

Another  feature  of  this  evaluation  was  its  use  of  historical  student 
records  as  a  source  of  data  for  some  of  the  study  groups.  Student  records  are 
maintained  at  ET  School  for  a  period  of  2  years  following  student  graduation.  By 
capitalizing  on  the  availability  of  such  information,  the  allocated  100 
experimental  students  were  reserved  for  the  study  of  key  training  issues;  there 
were  no  restrictions  on  the  numbers  of  conventionally  trained  students  available 
for  comparison.  Confidentiality  was  maintained  with  respect  to  all  student 
records. 


Data  Collect  ion 


The  data  collection  effort  necessary  to  support  the  TCT  was  concerned 
primarily  with  the  aggregation  and  refinement  of  existing  measures  rather  than 
the  development  of  new  performance  measures.  Student  aptitude,  prior 
achievement,  and  performance  measures  are  routinely  collected  by  ET  School 
personnel.  Data  related  to  equipment  reliability  and  maintainability  were  also 
tracked  by  school  personnel.  Data  collection  forms  introduced  by  DRI,  as  well  as 
those  already  in  use  by  the  ET  School,  are  discussed  below. 

A  second  aspect  of  the  data  collection  effort  included  a  review  of  the 
EEMT  laboratory  training  and  remediation  experiences  of  TCT  subjects.  Since 
this  information  was  stored  by  the  EEMT  system  on  individual  student  diskettes, 
an  appropriate  and  compatible  I/O  capability  was  required  to  collect  this 
information.  The  two  existing  instructor  terminals  were  already  extensively  used 
to  manage  student  training  and  to  perform  preventive  maintenance  functions.  An 
additional  terminal  was  acquired  for  the  collection  of  evaluation  data  to  minimize 
the  impact  of  the  TCT  data  collection  process  on  the  training  environment. 


Training  Effectiveness 

The  effectiveness  of  the  EEMT  system  for  training  ET  students  was 
assessed  primarily  through  the  examination  of  student  performance  measures. 
Student  scores  from  all  "A”  School  instructional  areas  were  transferred  from 
student  records  to  individual  ET  Student  Data  forms. 


Previous  Navy  guidelines,  including  the  contract  statement  of  work,  had 
suggested  the  use  of  pre-  and  postinstruction  measures  of  student  performance 
for  evaluating  training  effectiveness.  However,  this  approach  was  not 
particularly  well  suited  to  the  ET  School  environment.  First,  the  use  of  pre-  and 
posttest  measures  precluded  the  use  of  historical  student  data.  Second,  the  focus 
of  training  was  on  hands-on  proficiency.  For  safety  reasons,  a  hands-on  test 
could  not  be  administered  prior  to  adequate  familiarization  and  training  with 
operational  AN/SPS- 10  equipment.  In  lieu  of  pretesting,  the  TCT  included  the  use 
of  previous  aptitude  and  achievement  measures  to  ensure  that  different  training 
groups  were  equated  in  terms  of  prior  abilities. 

Scores  from  ET  School  PTs  were  the  principal  measure  of  EEMT  training 
effectiveness.  This  approach  to  performance  measurement  was  chosen  because 
(1)  the  nature  of  the  eventual  EEMT  intervention  was  uncertain  at  the  outset,  (2) 
PTs  were  administered  in  a  systematic  and  relatively  consistent  manner  for  all 
equipment  specific  training  areas,  (3)  PTs  were  administered  in  a  hands-on 
troubleshooting  format,  and  (4)  extensive  historical  performance  data  were 
available  concerning  each  possible  PT  problem. 

Limited  follow-up  data  were  also  collected  in  order  to  track  the  need  for 
retraining  subsequent  to  "A"  School.  An  EEMT  Follow-up  Survey  form  was  used 
for  this  purpose  (rf.  Section  A-l  of  Appendix  A). 

System  Operational  Suitability 

Certain  operational  suitability  information  was  routinely  collected 
on-site  at  the  ET  School  using  student  EEMT  Trainer  Critiques,  EEMT  Run-time 
Logs,  and  standard  ET  School  maintenance  forms  (2-KILO,  Daily  Production 
Reports  and  Down  Equipment  Logs).  The  EEMT  Trainer  Critique  form  used  at  the 
school  was  too  general  to  meet  the  needs  of  the  TCT.  A  more  detailed  DRI- 
generated  form  (rf.  Section  A-2  of  Appendix  A)  was  used  to  collect  specific 
student  feedback  about  the  operational  equipment  and  EEMT  trainers.  In 
addition,  two  new  data  collection  forms  were  used.  The  first,  an  EEMT  Training 
Availability  form,  was  designed  to  track  the  availability  of  EEMT  equipment  for 
training  (rf.  Section  A-3  of  Appendix  A).  The  second  form,  the  Instructor 
Critique,  was  used  to  determine  the  extent  to  which  instructors  were  willing  to 
use  and  accept  the  EEMT  system  as  an  integral  part  of  the  ET  training  program 
(rf.  Section  A-4  of  Appendix  A). 

Much  of  the  data  on  system  capabilities  was  obtained  during  on-site 
interviews  with  ET  School  personnel.  Since  this  information  was  collected  on  an 
ongoing  but  intermittent  basis,  a  structured  interview  guide  was  used  (rf.  Section 
A-5  of  Appendix  A). 


System  Costs 

System  cost  data  were  collected  on-site  at  both  the  Navy  Personnel 
Research  and  Development  Center  and  the  Great  Lakes  Naval  Training  Center. 
Cost  Worksheets  were  used  to  code  costs  related  to  the  development  and  use  of 
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the  EEMT  system.  These  forms  were  derived  from  the  cost  model  outlined  in  the 
TCT  plan  and  were  designed  to  record  expenditures,  personnel  time,  and  facility 
requirements  for  hardware,  software,  lessonware,  and  support.  Enlisted  personnel 
costs  were  derived  from  the  Life  Cycle  Navy  Enlisted  Billet  Costs—FY  81 
(Koehler  <5c  Turney,  1981). 


Evaluation  Questions 


The  series  of  studies  were  designed  to  address  a  wide  range  of  TCT 
issues.  The  evaluation  questions  listed  below  were  derived  primarily  from  the 
NDCP  and  the  DTEMP.  The  questions  have  been  grouped  to  correspond  to  the 
three  major  components  of  the  TCT:  training  effectiveness,  operational 
suitability,  and  cost  considerations.  In  some  cases,  these  evaluation  questions 
were  translated  to  more  testable  hypotheses  which  relate  to  specific  TCT  studies. 
These  hypotheses  are  presented  in  the  appropriate  sections  thoughout  this  report. 

With  respect  to  training  effectiveness,  the  evaluation  questions 
examined  were: 

•  Will  the  EEMT  system  improve  the  transition  between  the 
electronics  fundamentals  and  equipment-specific  segments 
of  ET  electronics  maintenance  training? 

•  Can  a  training  improvement  of  at  least  10  percent  be 
demonstrated  through  use  of  the  EEMT  system? 

•  Can  the  EEMT  system  maximize  the  effectiveness  of 
individual  hands-on  maintenance  training? 

•  Can  the  EEMT  system  accommodate  low,  middle,  and  high 
ability  students? 

•  Will  the  use  of  EEMT  training  result  in  a  greater  need  for 
refresher  training  at  "C”  School  than  conventional 
training? 

•  Do  the  skills  and  knowledge  learned  on  the  EEMT  system 
generalize  to  the  larger  population  of  equipment 
maintained  by  ETs? 

•  Can  the  EEMT  system  reduce  reliance  on  operational  prime 
equipment  in  ET  training? 

The  evaluation  questions  examined  related  to  operational  suitability  were: 

•  Are  the  planned  spares  maintained  on  hand  sufficient  to 
repair  all  critical/major  failures? 


•  Can  the  EEMT  be  shown  to  be  available  for  scheduled 
operation  a  minimum  of  nine-tenths  (0.9)  of  the  time? 

•  Does  the  EEMT  incorporate  basic  system  safety  principles? 

•  Does  the  EEMT  incorporate  person/equipment  interface 
features  that  maximize  efficient  and  safe  operation  and 
maintenance? 

•  Is  the  EEMT  system  easily  transported;  can  it  be  easily 
assembled  and  disassembled? 

•  Are  the  EEMT  systems  physically,  functionally,  and 
electrically  compatible  with  the  school  laboratory 
environment? 

•  Is  performance  data  from  the  EEMT  system  interoperable 
with  the  Navy  CMI  system  used  in  the  BE&E  School? 

•  Can  operators  of  the  EEMT  system  be  trained  with  no  more 
than  2  weeks  of  formal  training  plus  on-the-job  training 
(OJT);  maintenance  personnel  trained  in  no  more  than  4 
weeks  of  formal  training  plus  OJT? 

•  Can  the  EEMT  system  be  maintained  and  logistically 
supported  by  Navy  personnel  at  ET  School? 

•  Are  students  willing  to  accept  the  EEMT  system  as  a  viable 
means  of  providing  ET  Scncoi  training? 

•  Are  instructors  willing  ;e  accept  the  EEMT  system  as  a 
viable  means  of  providing  ET  School  training? 

•  Is  the  EEMT  system  capable  of  providing  hands-on  training 
of  electronic  maintenance  tasks  and  principles? 

•  Can  the  EEMT  system  be  used  to  train  in  vacuum  tube, 
solid  state,  and/or  large-scale  integrated  (LSI) 
technologies? 

•  Can  the  EEMT  system  be  used  to  provide  generic  systems 
training,  that  is,  training  on  tasks  and  equipment  common 
to  the  population  of  task':  and  equipment  used  by  ETs? 

•  Can  the  EEMT  system  adapt  to  changing  job  demands? 

•  Does  the  use  of  the  EEMT  system  allow  for  an  increase  in 
the  student/instructor  ratio? 

With  respect  to  cost,  the  evalu?..l  n  tjues~.uns  examined  were: 


•  Can  the  EEMT  system  provide  electronics  training  at  less 
than  the  cost  needed  to  acquire,  operate,  and  maintain 
conventional  hardware  to  accomplish  equivalent  training? 

•  What  specific  recurring  design-to-cost  goal  per  student 
station  can  be  met  (including  lessonware  and  course 
materials)? 

In  the  following  sections  of  this  report,  these  evaluation  questions  are 
discussed  in  conjunction  with  the  specific  evaluation  methodologies  used  and  the 
findings  obtained  in  the  three  major  components  of  the  TCT. 
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III.  TRAINING  EFFECTIVENESS 


This  section  describe^  three  TCT  studies  that  examined  the  training 
effectiveness  of  the  2D  EEMT.  The  first  relied  on  an  analysis  of  existing  ET 
School  student  records  to  study  possible  EEMT  training  effects  during  the  Primary 
Power  area  of  AN/SPS-10  radar  instruction.  The  second  used  traditional 
experimental  methods  to  also  study  the  effects  of  EEMT  training  during  Primary 
Power.  The  third  study  was  designed  to  examine  the  long-range  impacts  of  EEMT 
training. 


Study  1;  Analysis  of  rlistorical  Data 


Experimental  method  relies  on  comparisons  among  treatment  conditions 
in  which  subjects  are  treated  identically  except  for  the  factor  or  factors  under 
investigation.  Traditionally,  these  conditions  are  realized  through  the  use  of 
controlled  settings  and  random  assignment  of  subjects.  However,  through 
judicious  selection  of  available  data,  it  is  often  possible  to  approximate  an 
experimental  situation.  Study  1  was  designed  in  this  way  to  provide  an  initial 
assessment  of  the  training  effectiveness  of  the  2D  EEMT  equipment  through  a 
review  of  historical  ET  School  student  files. 

Specifically,  the  following  hypotheses  were  addressed: 

•  The  2D  EEMT  system  improves  the  performance  of  ET 
trainees  when  employed  for  laboratory  and/or  remedial 
instruction  of  electronics  maintenance. 

•  The  2D  EEMT  system  improves  performance  of  ET  trainees 
when  used  to  deliver  necessary  laboratory-related  remedial 
instruction  only. 

•  Use  of  the  2D  EEMT  system  results  in  a  10  percent 
increase  in  student  performance  as  compared  with 
conventional  training. 

•  The  2D  EEMT  system  improves  the  performance  of  low, 
middle,  and  high  ability  students. 

•  The  2D  EEMT  system  reduces  reliance  on  operational 
prime  equipment  in  ET  training. 


Method 

All  data  for  Study  1  were  collected  from  historical  student  records  that 
were  maintained  on-site  at  the  ET  Schooi.  To  ensure  comparability  of  previous 
training  experiences,  only  records  of  non-nuclear  ET  students  (603  V  designation) 
were  reviewed. 


Student  records  were  assigned  to  one  of  three  independent  treatment 
conditions  as  defined  by  the  time  periods  of  Primary  Power  training  shown  in 
Table  1.  These  three  time  periods  correspond  to  different  levels  of  operational 
and  2D  EEMT  equipment  availability  for  laboratory  and  extra  study/remedial 
training.  That  is,  students  whose  records  were  assigned  to  Condition  1  received 
training  on  operational  equipment  only.  During  the  time  period  for  Condition  2, 
operational  equipment  only  was  available  for  laboratory  training,  but  the  2D 
EEMT  equipment  was  available  for  extra  study/remediation.  During  the  time 
period  for  Condition  3,  both  operational  and  2D  EEMT  equipment  were  used  for 
laboratory  training;  the  2D  EEMT  was  also  available  for  extra  study/remediation. 
In  general,  students  were  trained  on  operational  AN/SPS-10  equipment  in  pairs  or 
groups  of  three;  EEMT  training  was  typically  conducted  individually,  although 
students  were  occasionally  trained  in  pairs  on  the  2D  EEMT. 

It  is  important  to  recognize  the  difference  between  the  assignment  of 
student  records  to  experimental  conditions  in  terms  of  equipment  availability,  as 
in  this  study,  and  assignment  on  the  basis  of  the  actual  types  of  training  received 
by  each  student  (as  in  Study  2).  Detailed  information  concerning  the  nature  of 
each  student's  Primary  Power  training  experiences  was  not  available  from 
historical  records.  It  is  known  that  laboratory  training  for  Conditions  1  and  2 
consisted  entirely  of  operational  equipment  training  and  that  for  Condition  3, 
operational  equipment  training  was  supplemented  by  limited  2D  EEMT  use. 
Typically,  use  of  the  2D  EEMT  consisted  of  a  brief  "light  off'  lesson  which 
reviews  system  turn-on  procedures,  a  "normal  operation"  lesson  during  which 
students  are  able  to  spend  an  unspecified  amount  of  time  examining  normal 
system  operating  conditions,  and  one  or  two  troubleshooting  lessons.  While  extra 
study/remediation  was  a  significant  component  of  training  for  most  students,  it  is 
not  known  to  what  extent  students  in  Conditions  2  and  3  made  use  of  the  2D 
EEMTs. 

Matched  samples  of  student  records  were  created  on  the  basis  of  student 
sex  and  ET  School  achievement  prior  to  radar  training.  The  resulting  sample 
sizes,  72  per  Condition,  are  also  shown  in  Table  1.  These  samples  were  then 
divided  into  three  equal  subsamples  on  the  basis  of  prior  achievement  to  indicate 
students  of  low,  middle,  and  high  ability.  Mean  cumulative  ET  School  grade 
averages  prior  to  radar  were  80.22,  83.96,  and  88.63,  respectively  for  the  three 
ability  levels.  Thus,  24  student  records  were  assigned  to  each  cell  of  the  resulting 
three  levels  of  equipment  availability  by  three  levels  of  ability  design. 


Results 

The  primary  measure  of  student  performance  was  the  PT  administered 
individually  to  all  students  at  the  end  of  Primary  Power  training.  This  test  is  a 
hands-on,  timed,  single  fault  troubleshooting  test  conducted  using  the  operational 
AN/SPS-10  equipment.  Any  one  of  20  different  problems  (faults)  can  be  selected 
for  a  particular  PT  although  the  same  problem  is  generally  administered  to  all 
students  on  the  same  training  shift.  While  these  problems  do  vary  in  difficulty,  an 


analysis  of  PT  records  showed  that  the  average  difficulty  of  each  test 
administered  was  relatively  stable  across  the  three  historical  time  periods  of 
interest A 

Figure  3  shows  the  PT  results  as  a  function  of  student  ability  levels  and 
training  equipment  availability.  Overall,  mean  PT  scores  were  78.89,  87.88,  and 
89.98  for  Conditions  1,  2,  and  3,  respectively.  Mean  scores  ranged  from  79.88  for 
low  ability  students,  to  86.12  for  middle  ability  students,  and  to  91.75  for  students 
of  high  ability.  A  complete  summary  of  PT  scores  is  provided  in  Table  B-l  of 
Appendix  B.  A  3x3  analysis  of  variance  (ANOVA)  revealed  significant  main 
effects  of  both  student  ability  [F(2,207)  =  10.05,  £<.001  ]  and  training  equipment 
availability  [  F(2,207)  =  8.07,  £  <  .001]  ;  the  interaction  between  these  variables 
was  not  significant.  Collapsing  over  ability  levels,  further  analysis  showed  that 
students  trained  during  time  periods  when  the  2D  EEMT  was  available  (Conditions 
2  and  3  combined)  performed  better  than  those  who  had  only  conventional 
AN/SPS-10  training  [t(  110)  =  3.81,  £  <  .001].  In  terms  of  mean  PT  scores,  this 
difference  represents  an  11.3  percent  increase  following  the  introduction  of  the 
2D  EEMT  equipment.  Furthermore,  this  effect  was  apparently  not  dependent  on 
the  level  of  student  ability  as  defined  by  prior  ET  School  achievement. 

Scores  from  the  Primary  Power  Unit  Test— a  paper-and-pencil,  multiple 
choice,  declarative  knowledge  test--were  also  examined.  A  3x3  ANOVA  showed  a 
significant  effect  of  student  ability  level  [F(2,2 07)  =  15.22,  £  <  .001] .  The  main 
effect  of  equipment  availability  and  of  ability  by  equipment  interaction  were  not 
significant.  This  finding  was  not  unexpected  in  view  of  the  troubleshooting  nature 
of  the  2D  EEMT  lessonware  and  the  factual  nature  of  the  Unit  Test. 


Discussion 

The  results  of  Study  1  indicated  that  use  of  the  2D  EEMT  improved  the 
performance  of  ET  School  trainees  as  measured  by  the  hands-on  troubleshooting 
test.  However,  since  both  EEMT  Conditions  (2  and  3)  performed  equally  well,  it  is 
tentatively  concluded  that  the  locus  of  this  effect  was  in  the  availability  of  the 
2D  EEMT  for  extra  study/remedial  training.  Introduction  of  the  EEMT  into  the 
Primary  Power  laboratory,  Condition  3,  did  not  result  in  an  additional 
performance  increment  as  compared  with  Condition  2.  The  most  likely 
explanation  for  the  superior  performance  of  students  in  Conditions  2  and  3  is  that 
the  2D  EEMT  provided  an  opportunity  for  additional  troubleshooting  training 
during  extra  study/remediation  that  was  not  available  to  conventionally  trained 
students.  Also,  since  the  EEMT  was  primarily  used  in  an  individual  training 
format,  students  who  trained  with  it  may  have  been  better  prepared  for  the 
individually  administered  PT. 

1  Estimates  of  difficulty  for  each  administration  were  computed  using 
the  mean  score  for  each  problem  based  on  all  tests  administered  to  date  and  the 
number  of  students  who  worked  each  problem  during  the  particular  time  period. 
The  resulting  estimates  were  83.3,  82.3,  and  82.3  for  Conditions  1,  2,  and  3, 
respectively.  It  was  not  possible  to  directly  associate  particular  students  with 
particular  Performance  Test  problems. 
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The  finding  that  students  trained  during  the  two  time  periods  when  the 
2D  EEMT  was  available  (Conditions  2  and  3)  performed  equally  well  suggests  that 
the  EEMT  can  reduce  reliance  on  operational  prime  equipment.  Since  the  only 
training  difference  between  these  Conditions  was  the  use  of  the  2D  EEMT  during 
the  AN/SPS-10  laboratory,  students  in  Condition  3  received  some  training  on  the 
2D  EEMTs  that  would  normally  have  been  provided  via  operational  equipment 
without  any  decrement  in  PT  performance.  It  must  be  considered,  however,  that 
the  extent  to  which  the  EEMTs  were  actually  used  for  laboratory  training  is 
unknown.  Thus,  the  extent  to  which  the  EEMTs  reduced  reliance  on  operational 
equipment  is  also  unknown,  although  it  is  assumed  to  be  relatively  slight. 

Finally,  the  results  support  the  hypothesis  that  the  2D  EEMT  improves 
the  performance  of  low,  middle,  and  high  ability  students.  Students  of  all  three 
ability  levels  were  affected  equally  by  the  availability  of  the  2D  EEMTs.  The 
next  study  in  this  section  was  conducted  to  provide  a  more  rigorous  examination 
of  the  effects  suggested  by  Study  1. 

An  additional  factor  to  be  considered  with  respect  to  these  findings  is 
the  substantial  student  backlog  problem  that  existed  at  ET  School  at  the  time  the 
EEMT  was  introduced.  In  fact,  acquisition  of  the  EEMT  units  was  in  part  a 
response  to  the  need  for  increased  student  flow  to  alleviate  the  backlog  and  the 
large  class  sizes  that  were  associated  with  it.  Since  large  class  sizes  may  have 
resulted  in  higher  than  optimal  student-to-equipment  ratios  in  the  training 
laboratory,  the  need  for  laboratory  related  extra  study  may  have  been  heightened, 
and  thus  the  impact  of  the  2D  EEMTs  exaggerated. 


Study  2:  Effectiveness  of  the  2D  EEMT  as  a  Supplement 


to  Operational  Equipment  Trainers 


The  overall  objective  of  this  study  was  to  determine  the  impact  of 
incorporating  the  2D  EEMT  into  the  existing  ET  School  laboratory  and  extra 
study/remediation  curricula.  Given  this  overall  objective,  the  study  addressed  the 
following  six  hypotheses: 

•  The  2D  EEMT  system  improves  the  performance  of  ET 
trainees  when  employed  for  laboratory  and  remedial 
instruction  of  electronics  maintenance  training. 

•  Use  of  the  2D  EEMT  system  results  in  a  10  percent 
increase  in  student  performance  as  compared  with 
conventional  training. 

•  The  2D  EEMT  system  can  maximize  the  effectiveness  of 
individual  hands-on  maintenance  training. 

•  The  2D  EEMT  system  can  maximize  the  effectiveness  of 
hands-on  maintenance  training  when  delivered  using  the 
traditional  student  team  approach. 


•  The  2D  EEMT  system  improves  the  performance  of  low, 
middle,  and  high  ability  students. 

•  The  2D  EEMT  system  reduces  reliance  on  operational 
prime  equipment  in  ET  training. 


Method 

Both  the  training  mode  and  format  of  Primary  Power  training  were 
manipulated  for  students  participating  in  Study  2  of  the  TCT.  The  resulting 
experimental  design  is  illustrated  in  Figure  4. 


Training 

Format 

Training". 

Mode  X. 

Individual 

Students 

Pairs  of 
Students 

High  EEMT 
Supplement 

Group  IH 

Group  PH 

Low  EEMT 
Supplement 

Group  IL 

Group  PL 

Operational 

Only 

Group  IO 

Group  PO 

Figure  4.  Experimental  groups  for  Study  2. 


Training  modes.  Student  training  modes  were  defined  by  the  extent  to 
which  2D  EEMT  equipment  was  used  to  supplement  a  student's  laboratory  and 
extra  study/remediation  training.  Three  modes  were  selected  for  study:  an 
Operational  Only  mode  in  which  all  laboratory  training  was  conducted  using 
operational  AN/SP5-10  equipment,  a  Low  EEMT  mode  in  which  the  EEMT  was 
used  for  approximately  30  percent  of  laboratory  training,  and  a  High  EEMT  mode 
in  which  the  EEMT  was  used  for  approximately  60  percent  of  laboratory  training. 
The  total  amount  (hours)  of  laboratory  training  was  the  same  for  all  these  modes. 
Students  in  the  High  and  Low  EEMT  conditions  were  also  allowed  to  use  the  EEMT 
for  extra  study;  students  in  the  Operational  Only  condition  used  conventional 
workbooks  for  extra  study  training. 


Training  time  on  operational  equipment  was  monitored  and  controlled  by 
ET  School  personnel  and  is  summarized  in  Table  2  for  the  three  training  modes 
used  in  Study  2.  Overall,  High  EEMT  students  averaged  2.78  hours  and  Low  EEMT 
students  averaged  4.59  hours  of  training  on  operational  AN/SPS-10  equipment 
during  the  7-8  hours  of  laboratory  training.  Although  Table  2  shows  that 
Operational  Only  students  averaged  6.24  hours  on  operational  equipment,  this 
figure  is  an  underestimate  because  student  training  times  were  monitored  only  up 
to  7  hours.  Thus,  actual  training  times  were  relatively  consistent  with  those 
specified  by  the  research  design  for  High  EEMT,  Low  EEMT,  and  Operational  Only 
modes. 


The  number  of  problems  completed  by  students  in  the  different  training 
conditions  was  also  monitored  by  ET  School  personnel  and  is  summarized  in  Table 
B-2  of  Appendix  B.  Because  the  solution  times  for  training  problems  varied 
widely  across  both  problems  and  students,  each  training  mode  actually  included  a 
range  of  troubleshooting  experience.  Overall,  students  assigned  to  EEMT 
conditions  completed  slightly  more  problems  during  laboratory  training  (8.30  on 
the  average)  than  those  assigned  to  Operational  Only  training  (7.55).  In  addition, 
EEMT  students  completed  an  average  of  1.16  problems  during  extra 
study/remediation  while  students  in  the  Operational  Only  conditions  completed 
only  workbook  exercises. 

Training  formats.  Two  different  training  formats  were  employed, 
individual  and  pair.  Students  trained  in  an  individual  format  worked  all  laboratory 
and  extra  study  problems  alone;  those  assigned  to  the  pair  format  worked  all 
laboratory  problems  with  another  student  but  were  allowed  to  attend  extra  study 
on  their  own.  Students  trained  in  either  format  were  provided  equivalent  amounts 
of  operational  equipment  training  time  overall.  As  shown  in  Table  2,  actual 
operational  training  times  did  not  differ  appreciably  between  students  in 
individual  and  pair  formats,  averaging  4.40  hours  and  4.20  hours,  respectively. 
However,  students  assigned  to  the  pair-training  format  were  able  to  complete 
slightly  more  problems,  both  on  EEMT  and  operational  equipment,  than  those 
assigned  to  the  individual  format  as  shown  in  Table  B-2  of  Appendix  B. 

Subjects.  A  total  of  157  non-nuclear  ET  students  was  randomly  assigned 
to  one  of  six  experimental  conditions  defined  by  crossing  the  three  levels  of 
training  mode  (High  EEMT,  Low  EEMT,  and  Operational  Only)  with  the  two 
training  formats  (individual  and  pair).  Assignments  were  mixed  within  classes  so 
that  the  effects  of  instructors,  training  shifts,  and  test  problems  were 
counterbalanced.  Subsequent  to  ET  School  graduation,  each  student's  academic 
record  was  also  reviewed  to  obtain  information  related  to  student  ability. 


Results 


Even  though  students  were  assigned  randomly  to  experimental 
conditions,  the  question  of  whether  students  in  various  conditions  were  of  equal 
ability  was  examined  to  ensure  uniformity  of  group  characteristics  to  the  research 
design.  Cumulative  ET  School  achievement  scores  prior  to  radar  training  were 
submitted  to  a  3x2  ANOVA  (three  levels  of  training  mode  by  two  levels  of  training 
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Table  2 


Laboratory  Training  Received  on  Operational  Equipment 
as  a  Function  of  Training  Mode  and  Format 


Training 

Training  Format 

Mode 

Individual 

Pair 

Overall 

High  EEMT 

N 

27 

24 

51 

Supplement 

X 

2.74 

2.84 

2.78 

a 

0.43 

0.25 

0.36 

Low  EEMT 

N 

28 

27 

55 

Supplement 

X 

4.81 

4.36 

4.59 

a 

0.93 

0.88 

0.92 

Operational 

N 

16 

16 

32 

Equipment  Only 

X 

6.50* 

5.98* 

6.24* 

o 

0.67 

1.09 

0.93 

Overall 

N 

71 

67 

138 

X 

4.40* 

4.20* 

4.30* 

o 

1.62 

1.43 

1.53 

♦Because  training  time  was  only  aggregated  to  a  maximum  of  7 
hours,  these  values  may  be  slightly  underestimated. 


format).  No  significant  differences  among  groups  were  found.  Thus,  students 
assigned  to  each  of  the  six  conditions  were  of  comparable  ability. 

Performance  test  results.  PT  scores  were  the  primary  measure  of 
training  effectiveness  and  are  summarized  in  Table  3  as  a  function  of  training 
mode  and  format.  Mean  scores  were  82.59,  78.63,  and  80.11  for  High  EEMT,  Low 
EEMT,  and  Operational  Only  conditions,  respectively;  79.64  for  individuals  and 
81.11  for  pairs.  A  3x2  ANOVA  conducted  on  these  data  revealed  no  significant 
main  effect  of  either  factor  and  no  significant  mode  by  format  interaction. 

Times  to  completion  for  PTs  as  shown  in  Table  4,  were  also  analyzed. 
Mean  times  to  completion  were  31.70,  35.66,  and  29.57  minutes  respectively  for 
High  EEMT,  Low  EEMT,  and  Operational  Only  conditions.  Individually  trained 
students  took  an  average  of  31.41  minutes  and  pair- trained  students  took  an 
average  of  34.19  minutes.  When  these  data  were  submitted  to  a  3x2  ANOVA,  a 
significant  main  effect  of  equipment  mode  [F(2, 141)  =  3.76,  £  <.05  ]  and  a 
marginally  significant  effect  of  training  format  [F(l,  141)  =  3.41,  £  <.l  ]  were 
found.  The  mode  by  format  interaction  did  not  approach  significance. 

Closer  examination  of  the  data  in  Table  4  suggested  that  these  effects 
might  be  accounted  for  by  the  particularly  slow  PT  completion  times  of  students 
assigned  to  Group  PL  (rf.  Figure  4).  Further  comparisons  confirmed  that  (1) 
Group  PL  students  were  significantly  slower  than  those  in  Group  IL  [t(51)  =  2.13, 
£  <  .05  ]  and  (2)  Group  PL  students  were  significantly  slower  than  students  in 
Groups  PH  and  PO  combined  [t(70)  =  3.04,  £  <  .01]  .  However,  the  reason  for  the 
unusually  slow  performance  of  Group  PL  students  is  unclear.  One  possibility  is 
that  because  the  2D  EEMT  was  designed  as  an  individual  trainer,  it  may  not  be 
well  suited  to  the  pair  training  format  and  students  who  used  it  in  pairs  were  at  a 
disadvantage.  However,  this  explanation  would  suggest  that  students  in  Group  PH 
would  be  even  more  disadvantaged  since  they  received  more  pair  training  on  the 
2D  EEMT.  This  was  not  found  to  be  the  case.  Another,  and  more  likely, 
possibility  is  simply  that  EEMT  training  modes  and  the  pair  format  are  the 
conditions  that  are  least  similar  to  the  PT  testing  conditions.  Unfortunately,  this 
does  not  explain  why  Group  PL  students  performed  slower  than  Group  PH  students 
either.  One  final  possibility  is  that  familiarization  with  the  EEMT  equipment  took 
longer  for  pairs  than  individuals.  If  this  was  the  case,  and  it  is  assumed  that  truly 
effective  training  only  takes  place  once  students  are  familiar  with  a  training 
device,  it  could  be  argued  that  students  were  not  given  sufficient  EEMT 
experience  in  the  Low  EEMT  condition  for  optimum  training  effectiveness. 

Overall,  then,  while  troubleshooting  accuracy  was  not  influenced  by  the 
equipment  training  mode  or  training  format,  the  speed  of  troubleshooting  was 
somewhat  affected.  Specifically,  students  trained  on  operational  equipment 
appeared  faster  than  those  who  received  2D  EEMT  training,  and  students  trained 
as  individuals  were  slightly  faster  than  those  trained  in  pairs. 

Instructor  ratings.  Instructor  ratings  of  student  AN/SPS -10  laboratory 
performance  (rf.  Table  5)  were  also  collected  (using  a  four-point  scale)  and 
analyzed  using  the  3x2  ANOVA  approach.  A  marginally  significant  main  effect  of 
equipment  training  mode  f  F(2, 1 3 1)  =  2.47,  £<  ,1J  was  revealed.  Further  analysis 


Table  3 


Performance  Test  Scores  as  a  Function 
of  Training  Mode  and  Format 


Training 

Mode 

Training  Format 
Individual  Pair 

Overall 

High  EEMT 

N 

25 

22 

47 

Supplement 

X 

82.51 

82.68 

82.59 

a 

15.02 

14.77 

14.74 

Low  EEMT 

N 

26 

27 

53 

Supplement 

X 

79.99 

77.32 

78.63 

a 

20.44 

26.22 

23.38 

Operational 

N 

23 

24 

47 

Equipment  Only 

X 

76.11 

83.93 

80.11 

o 

23.97 

18.86 

21.64 

Overall 

N 

74 

73 

147 

X 

79.64 

81.11 

80.37 

a 

19.93 

20.84 

20.33 

23 


Table  4 

Times  to  Completion  (in  Minutes)  for  Performance  Test 
as  a  Function  of  Training  Mode  and  Format 


Training 

Mode 


Training  Format 
Individual  Pair 


Overall 


High  EEMT 

N 

25 

22 

4 

7 

-  w  •  • 

Supplement 

X 

30.80 

32.73 

3 

1 

.70 

0 

9.53 

11.09 

1 

0 

.22 

Low  EEMT 
Supplement 


Operational 
Equipment  Only 


Overall 


26 

32.04 

13.11 


23 

29.17 

12.67 


24 

29.96 

9.38 


51 

35.66 

12.56 


47 

29.57 

11.00 


1 

s 

f: 

Low  EEMT 
Supplement 

N 

X 

o 

25 

2.12 

0.97 

23 

1.74 

0.86 

48 

1.94 

0.93 

t 

V* 

N 

o 

»*, 

• 

1 

r» 

r. 

Operational 

N 

22 

23 

45 

V 

| 

Equipment  Only 

X 

1.91 

1.35 

1.62 

* 

• 

: 

r, 

a 

1.11 

0.49 

0.89 

■w- 

66 

1.61 

0.68 


137 

1.87 

0.91 


•Rating  Scale:  1  =  top  quarter  of  students,  2  =  second 
quarter,  3  =  third  quarter,  4  =  last  quarter. 
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showed  that  students  who  trained  only  on  operational  equipment  were  rated  as 
superior  (mean  rating  =  1.62)  to  those  who  trained  on  the  2D  EEMT  (mean  rating 
for  High  EEMT  and  Low  EEMT  students  combined  =  1.99).  The  ANOVA  also 
revealed  a  significant  main  effect  of  training  format 
[  F(  1 , 1 3 1 )  =  10.72,  £  <.001  ]  .  Students  who  worked  in  pairs  were  rated  superior  in 
performance  to  those  who  worked  individually  (mean  ratings  =  1.61  and  2.11, 
respectively).  The  interaction  of  training  mode  and  format  was  not  significant. 
Thus,  instructor  ratings  were  highest  for  students  assigned  to  conventional 
training  conditions,  i.e.,  operational  equipment  mode  and  pair  format. 

Training  effects  and  student  ability.  In  order  to  examine  the  effects  of 
2D  EEMT  training  on  low,  middle,  and  high  ability  students,  equal  numbers  of 
students  were  assigned  to  each  of  the  three  ability  levels  on  the  basis  of  prior  ET 
School  achievement  as  in  Study  1.2  Assignments  were  collapsed  across  training 
formats  and  Primary  Power  PT  scores  were  then  analyzed  in  the  resulting 
equipment  mode  by  ability  level  framework.  Table  B-3  in  Appendix  B  summarizes 
the  findings.  A  3x3  ANOVA  revealed  a  significant  main  effect  of  ability 
level  [  FC 2, 1 3 S)  =  8.08,  £  <  .00 1  ]  but  no  significant  effect  of  training  mode  and  no 
significant  interaction.  Thus,  while  PT  scores  varied  as  a  function  of  ability  level, 
the  PT  scores  of  students  at  different  ability  levels  were  not  affected  by  training 
modes. 

A  similar  analysis  of  PT  completion  times  was  also  conducted.  In 
addition  to  the  marginally  significant  effect  of  training  mode  noted  earlier,  no 
significant  effects  of  ability  level  or  ability  by  mode  interaction  were  found. 
Apparently  PT  completion  times  were  more  closely  related  to  familiarity  with  the 
testing  conditions  than  to  students  ability  levels. 

Finally,  instructor  ratings  were  submitted  to  a  3x3  ANOVA  which  showed 
significant  main  effects  of  equipment  mode,  as  discussed  previously,  and  ability 
level  [F(2, 128)  =  3.27,  £  <  .05].  Again,  the  mode  by  ability  interaction  was  not 
significant. 

In  summary,  then,  while  all  three  performance  measures  were  related  to 
student  ability,  the  performance  of  students  at  each  level  was  not  differentially 
affected  by  the  extent  to  which  their  training  included  use  of  the  2D  EEMT 
equipment. 


Discussion 

Unlike  Study  1,  this  study  found  no  appreciable  differences  among 
students'  PT  performance  as  a  function  of  training  mode.  The  hypothesis  that 

2The  mean  cumulative  ET  School  averages  for  low,  middle,  and  high 
ability  students  were  84.61,  88.00,  and  91.76,  respectively.  While  these  scores 
were  somewhat  higher  than  the  corresponding  scores  from  Study  l,  this  difference 
may  have  resulted  from  changes  in  PT  scoring  methods  rather  than  qualitative 
changes  in  student  ability. 
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EEMT  training  improves  performance  was  not  supported.  Because  of  the 
historical  nature  of  Study  1,  certain  training  and  performance  data  were  not 
available  that  may  have  helped  resolve  this  apparent  discrepancy  in  training 
effectiveness.  For  example,  training  format,  extent  of  laboratory  and  extra  study 
EEMT  use,  and  PT  completion  times  may  have  differed  substantially  between 
Study  1  and  Study  2  students.  Another  possible  explanation  follows  from  the 
suggestion  that  the  effects  of  EEMT  training  in  Study  1  might  have  been 
magnified  because  of  the  student  backlog  problem.  Since  training  time  was 
controlled  in  Study  2,  no  similar  effect  would  be  anticipated. 

Although  training  effects  were  observed  for  PT  completion  times  in 
Study  2,  they  were  primarily  attributable  to  the  slow  performance  of  students 
assigned  to  the  Low  EEMT/pair  conditions.  The  reason  for  this  is  not  clear,  but  is 
most  likely  related  to  a  familiarity  with  the  PT  testing  conditions.  That  is, 
students  who  had  more  experience  working  on  the  operational  equipment  or 
working  as  individuals  were  faster  in  the  individual  operational  equipment  testing 
situation. 

Instructor  ratings  clearly  favored  students  in  conventional  training 
conditions  (operational  equipment  mode  and  pair  format)  over  those  in  EEMT 
training  conditions.  However,  these  data  may  reflect  a  preference  for  existing 
training  practices  or  perhaps  an  inability  to  make  accurate  ratings  under 
unfamiliar  training  conditions. 

The  findings  of  Study  2  do  suggest  that  the  2D  EEMT  can  reduce  reliance 
on  operational  equipment  in  ET  training  when  used  as  a  training  supplement. 
Specifically,  based  on  the  High  EEMT  conditions,  it  appears  that  half  of  the 
operational  equipment  laboratory  training  can  be  replaced  by  2D  EEMT  training 
without  a  decrement  in  student  troubleshooting  accuracy  as  long  as  EEMTs  are 
als£  available  for  extra  study/remediation  purposes. 


Study  3:  Follow-up  of  EEMT -Trained  Students 


The  objective  of  this  study  was  to  examine  the  long-range  impacts  of 
EEMT  "A”  School  training.  Specifically,  this  follow-up  effort  focused  on  the  need 
for  refresher  training  during  "C"  School  or  fleet  assignments  and  the 
generalizability  of  EEMT  training  to  the  larger  population  of  electronic 
equipment.  The  primary  hypotheses  to  be  addressed  are: 

•  Use  of  the  EEMT  for  ET  "A"  School  training  does  not  result 
in  subsequent  increased  refresher  training  needs. 

•  The  skills  and  knowledge  learned  on  the  EEMT  system 
generalize  to  the  larger  population  of  equipments 
maintained  by  ETs. 

The  ability  of  a  follow-up  study  to  detect  long-range  EEMT  effects  is 
related  to  two  factors:  the  strength  of  the  EEMT  training  intervention  and  the 
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length  of  time  between  that  training  and  the  follow-up.  Thus,  this  study  was 
designed  to  focus  on  students  who  received  the  greatest  amounts  of  EEMT 
training  and  examined  immediate  as  well  as  long-range  impacts. 


Method 

Students  who  participated  in  Study  2  also  served  as  the  subjects  for  this 
experiment.  Immediate  refresher  training  needs  (i.e.,  "A"  School  remediation  and 
setbacks)  were  tracked  as  part  of  the  review  of  student  ET  School  records.  A 
sample  of  the  students  was  also  tracked  to  "C"  School  or  advanced  training 
programs.  Due  to  the  large  number  of  possible  "C"  School  assignments,  only  those 
programs  to  which  a  significant  number  of  students  were  assigned  were  contacted. 
Performance  at  the  " C "  School  assignment  was  assessed  after  a  1  to  2  month 
period  of  advanced  training.  A  one-page  questionnaire  (see  Section  A-l  of 
Appendix  A)  was  distributed  to  the  immediate  training  supervisor  of  each  of  the 
technicians. 


Results 

"A"  School  refresher  training  needs.  ET  students  who  fail  a  PT  are 
assigned  to  compulsory  remediation  prior  to  retaking  the  failed  test.  The 
proportion  of  students  who  fail  the  Primary  Power  and  subsequent  radar  PTs, 
therefore,  is  a  direct  measure  of  the  need  for  "A"  School  refresher  training. 
These  values  were  computed  using  scores  obtained  from  student  records  and  are 
summarized  in  Table  6.  Overall,  the  proportion  of  students  requiring  compulsory 
laboratory-related  remediation  was  relatively  low  (less  than  .20)  during  each  of 
the  four  radar  training  areas. 

After  combining  the  two  EEMT  training  conditions,  these  data  were 
submitted  to  independent  2x2  Chi-Square  analyses  for  each  training  area.  No 
significant  differences  in  the  proportions  of  students  assigned  to  compulsory 
remediation  were  found  as  a  function  of  EEMT  vs.  conventional  training. 

"C"  School  follow-up.  Follow-up  questionnaires  were  completed  by  "C" 
School  supervisors  for  98  students  who  had  participated  in  Study  2.  Table  B-4  of 
Appendix  B  summarizes  their  ratings  with  respect  to  the  seven  questionnaire 
items  for  students  in  each  of  the  three  "A"  School  training  modes.  Mean  ratings 
were  slightly  above  average  for  every  item  (the  range  was  from  3.30  to  3.39)  and 
did  not  differ  as  a  function  of  training. 


Discussion 

The  follow-up  of  EEMT-trained  students  supported  the  hypothesis  that 
use  of  the  trainer  does  not  result  in  subsequent  increased  refresher  training  needs 
either  at  the  "A”  or  "C"  School  levels.  This  was  not  surprising  in  view  of  the 


Table  6 


Proportion  of  Students  Assigned  to  Compulsory  Laboratory- 
Related  Remediation  During  Radar  Training 


Training  Area 

TCT  Training  Mode 

High  EEMT 
(N=43)» 

Low  EEMT 
(N=51)» 

Operational  Only 
(N=49)* 

Primary  Power 

.05 

.14 

.10 

Transmitters 

.16 

.20 

.18 

Receivers 

.14 

.20 

.15 

AN/SPA-25 

.14 

.13 

.20 

♦These  values  vary  slightly  for  some  cells  of  the  matrix  because  of 
attrition  and  missing  data. 

relatively  minor  training  manipulations  employed  in  Study  2.  Comments  from  the 
follow-up  questionnaires,  structured  interview  findings,  and  comments  from 
student  follow-up  interviews^  supported  the  hypothesis  that  the  tasks,  skills,  and 
knowledge  learned  on  the  EEMT  system  generalize  to  the  larger  population  of 
equipment  maintained  by  ETs,  at  least  to  the  same  extent  as  those  learned  on 
operational  AN/SPS-10  equipment.  A  transfer  of  training  paradigm  would  be 
needed  to  address  this  issue  of  generalizability  in  a  more  direct  and 
comprehensive  fashion. 


Summary 


Analysis  of  the  training  effectiveness  of  the  2D  EEMT  both  during  and 
after  the  Primary  Power  area  of  AN/SPS-10  radar  instruction  indicates: 

•  When  used  for  a  limited  laboratory  and  extra  study 
instruction,  the  2D  EEMT  does  not  significantly  improve 
the  performance  of  ET  trainees. 

^Informal  "C"  School  interviews  were  conducted  at  Lowry  Air  Force 
Base,  Denver,  Colorado,  with  a  small  sample  of  TCT  participants. 
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•  When  used  in  conjunction  with  operational  equipment,  the 
2D  EEMT  system  can  reduce  reliance  on  operational 
equipment  by  50  percent  in  a  training  laboratory  setting. 

•  Use  of  the  2D  EEMT  is  equally  effective  for  low,  middle, 
and  high  ability  students. 

•  The  2D  EEMT  is  an  effective  alternative  to  the  use  of 
operational  equipment  for  providing  extra  study  and 
remediation  opportunities. 

•  Use  of  the  2D  EEMT  for  ET  "A"  School  training  does  not 
result  in  subsequent  increased  refresher  training  needs. 

•  The  skills  and  knowledge  learned  on  the  EEMT  system 
generalize  to  the  larger  population  of  equipment 
maintained  by  ETs. 
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IV.  OPERATIONAL  SUITABILITY 


The  evaluation  plan  of  the  operational  suitability  of  the  2D  EEMT  system 
for  ET  "A"  School  training  was  developed  to  address  a  wide  range  of  system- 
related  issues:  reliability,  availability,  maintainability,  appropriateness  for 
Primary  Power  training,  adaptability  to  other  training  areas,  and  user  acceptance. 
Specific  hypotheses  addressed  include: 

•  There  are  a  sufficient  number  of  spares  to  repair  all 
critical  or  major  failures  during  the  evaluation  period. 

•  The  EEMT  is  available  for  training  at  least  90  percent  of 
the  time. 

•  The  EEMT  incorporates  basic  safety  principles. 

•  Person-equipment  interface  features  maximize  efficient 
safe  operation  and  maintenance  of  the  2D  EEMT. 

•  The  2D  EEMT  system  is  easily  transported,  assembled  and 
disassembled. 

•  The  2D  EEMT  system  is  compatible  with  the  ET  "A"  School 
environment. 

•  Performance  data  from  the  2D  EEMT  system  are 
interoperable  with  the  Navy  CMI  system. 

•  The  2d  EEMT  can  adapt  to  changing  job  demands. 

•  Training  requirements  for  2D  EEMT  operations  and 
maintenance  personnel  are  within  specified  limits. 

•  The  2D  EEMT  can  be  used  to  train  to  appropriate  vacuum 
tube,  solid  state,  and  LSI  technology. 

•  The  2D  EEMT  can  be  maintained  and  logistically  supported 
by  Navy  personnel  at  ET  "A"  School. 

•  The  2D  EEMT  will  allow  an  increase  in  the  student- 
instructor  ratio. 


Reliability 


Unscheduled  maintenance  activity  was  tracked  to  determine  the 
reliability  level  of  the  2D  EEMT  units.  Reliability  was  defined  as  the  extent  to 
which  the  2D  EEMT  units  allocated  to  the  Primary  Power  training  required 
unscheduled  maintenance  during  the  TCT  period.  The  approach  taken  to  measure 


the  reliability  of  the  units  included  in  the  study  was  to  review  the  ET  "A"  School 
Support  Division  Daily  Production  Reports  (DPR)  for  the  period  13  June  1983 
through  9  September  1983,  the  DRI  Training  Availability  forms,  the  Support 
Division  Down  Equipment  Daily  Logs  and  structured  interviews  with  ET  "A” 
School  personnel  (rf.  Section  A-4  of  Appendix  A). 

The  DPRs  for  this  period  revealed  that  the  18  units  assigned  to  Primary 
Power  training  (laboratory  and  extra  study/remediation)  required  a  total  of  118 
individual  repair  actions.  The  actions  counted  do  not  include  scheduled  equipment 
maintenance,  failures  corrected  by  laboratory  instructors  at  the  time  of  the 
event,  or  instances  in  which  malfunctions  were  reported  by  laboratory  instructors 
but  could  not  be  duplicated  by  the  maintenance  technician.  To  provide  a  more 
detailed  account  of  system  reliability,  the  frequency  of  various  types  of  problems 
is  presented  in  Table  7. 

One  problem  identified  by  ET  "A"  School  personnel  and  students,  but  not 
reflected  in  maintenance  data,  was  the  EEMT  system  lockup.  During  lesson 
operation,  the  2D  EEMT  would  "freeze"  at  a  random  point  in  the  lesson.  While  the 
system  could  be  restarted  by  the  instructor,  student  progress  in  troubleshooting 
and  the  student  performance  record  were  lost.  Since  the  lockup  problem  did  not 
require  maintenance  personnel  intervention,  incidents  of  system  lockup  were  not 
recorded  on  DPRs.  Although  the  frequency  of  system  lockup  was  not  recorded, 
the  problem  was  considered  particularly  frustrating  (as  reflected  in  the  User 
Acceptance  analysis)  due  to  the  need  to  restart  and  the  loss  of  the  student 
performance  record.  Support  Division  personnel  suggested  heat  buildup  within  the 
2D  EEMT  system  as  a  possible  cause  of  the  lockup  problem.  The  placement  of  the 
videodisc  player  and  the  microprocessor  in  the  console  cabinet  (rf.  Figure  1)  did 
result  in  heat  buildup  due  to  insufficient  air  ventilation/circulation.  Nevertheless, 
no  direct  connection  was  made  during  the  TCT  between  heat  buildup  and  system 
lockup.  These  problems  were  being  addressed  by  manufacturer  and  Support 
Division  personnel  at  the  end  of  the  T CT. 

Maintenance  Material  Management  (3M)  reports  for  the  AN/SPS-10 
Primary  Power  laboratory  equipment  for  the  report  period  January  1983  through 
September  1983  were  reviewed.  They  showed  16  repair  actions  to  this  operational 
equipment.  Ail  repair  actions  to  AN/SPS-10  equipment  are  the  responsibility  of 
RCA  personnel. 


Summarv 


The  results  of  the  equipment  reliability  analysis  indicate? 

•  Unscheduled  maintenance  actions  on  the  18  2D  EEMT  units 
numbered  118  during  the  TCT.  A  total  of  16  unscheduled 
maintenance  actions  on  the  operational  AN/SPS-10  were 
required  during  a  somewhat  longer  period  of  time. 

•  Maintenance  actions  related  to  touch  panels  for  the  2D 
EEMT  units  included  in  the  TCT  accounted  for  a  total  of  46 
of  118,  or  40  percent,  of  recorded  maintenance  actions 
during  the  study  period. 
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Table  7 

2D  EEMT  Maintenance  Actions 
(13  3une  to  9  September  1983) 


Number  of 

Component/ Action 

Occurrences 

12"  CRT: 

♦Retape  touch  panel 
♦Replace  touch  panel 
Calibrate  x,y  origin 
and  gain 

19"  CRT: 

♦Retape  touch  panel 
♦Replace  touch  panel 
Calibrate  x,y  origin 
and  gain 


Repair  chassis 

Videodisc  Player: 

Repair 

Altos  Microprocessor: 

Replace  CPU  board 
Replace  video  RAM 
Replace  PIA  board 
Replace  boot  switch 
Replace  disc  drive  cable 
Unspecified  malfunction 

Miscellaneous: 

♦Constant  contact  on  touch  panel 
Adjust  screen  voltage  and  bias 
Replace  cable  between  LDP  and  Altos 
Check  operation  and  calibration 
♦Check  touch  panel  alignment 
Adjust  color  and  brightness 
♦Adjust  reference 
Intermittent  lockup 


Total  repair  actions 


♦Related  to  touch  panel  maintenance  actions.  Total  number 
equals  46,  or  40  percent  of  total  2D  EEMT  maintenance  actions. 
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Availability 


An  important  element  in  the  evaluation  of  a  training  device  is  the 
frequency  with  which  it  is  available  for  training  when  needed.  Training 
availability  was  assessed  by  review  of  the  ET  "A"  School  Support  Division  Down 
Equipment  daily  logs  and  by  analysis  of  the  Training  Availability  forms.  Down 
Equipment  Daily  Logs  list  all  2D  EEMT  components  which  are  inoperable  by  room 
location,  problem  and  anticipated  date  of  repair. 

Results  of  the  analysis  are  shown  in  Figure  5.  Specifically,  in  the  period 
of  time  during  which  equipment  downtime  was  tracked  (13  3une  through  9 
September),  there  were  1,224  total  unit-operating  days  (68  classroom  days  x  18 
units).  Of  these,  there  were  338  down  equipment  days  attributable  to  the  18  2D 
EEMT  units.  The  Down  Equipment  Daily  Logs  show  that  the  8  2D  EEMT  units 
assigned  to  the  Primary  Power  laboratory  accounted  for  23  down  equipment  days. 
This  results  in  4  percent  downtime  for  the  units  specifically  assigned  to  this 
laboratory,  or  a  96  percent  availability. 

Component  swapping  among  2D  EEMT  units  was  possible  and  frequently 
occurred,  meaning  a  system  could  be  operating  even  if  one  of  its  original 
components  was  on  the  Down  Equipment  logs.  Therefore,  a  more  accurate 
measure  of  Primary  Power  laboratory  trainer  availability  was  the  Training 
Availability  forms.  Primary  Power  laboratory  instructors  were  asked  to  indicate 
on  the  Training  Availability  forms  whether  the  2D  EEMT  units  were  available  for 
training  during  their  shift  on  each  day  of  laboratory  training.  Results  of  the 
analysis  of  the  information  contained  on  the  Training  Availability  forms  during 
the  period  of  the  TCT  reveal  that  the  units  allocated  to  the  Primary  Power 
laboratory  were  available  for  training  90.8  percent  of  the  time.  Specifically,  the 
eight  units  in  the  Primary  Power  laboratory  were  tracked  for  384  unit-training 
sessions  over  the  12  weeks  of  the  TCT  (8  units  x  3  training  days  per  shift  x  16 
shifts  =  384  unit-training  sessions)^.  Downtime  accounted  for  35.5  unit-training 
sessions,  or  9.2  percent.  The  Training  Availability  forms  also  showed  that  only 
one  2D  EEMT  unit  of  the  8  units  assigned  to  the  Primary  Power  laboratory  was 
inoperable  for  an  extended  period  (7  weeks,  or  over  50  percent  of  the  TCT  period), 
while  two  of  the  10  2D  EEMT  units  assigned  to  Primary  Power  extra 
study/remediation  were  down  for  the  whole  TCT  period,  and  seven  of  the  10  were 
down  at  the  end  of  the  TCT  period. 

Operational  equipment  availability  for  training  during  the  TCT  was  also 
tracked  on  the  Training  Availability  forms.  Only  seven  operational  equipment 
unit-training  sessions  were  missed  over  480  total  unit-training  sessions  (10  units  x 
3  training  sessions  per  shift  x  16  shifts  =  480  unit-training  sessions).  This  results 
in  a  98.5  percent  availability  rate.  None  of  the  AN/SPS-10  units  assigned  to  the 
Primary  Power  laboratory  were  down  for  an  extended  period  of  time. 

^Since  all  three  shifts  of  each  class  were  not  necessarily  included  in  the 
TCT,  the  shift  has  been  used  as  the  unit  of  measure. 
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Summary 

The  results  of  the  availability  analysis  indicate  that: 

•  The  2D  EEMT  units  assigned  to  Primary  Power  training 
were  available  72  percent  of  the  required  time.  This  did 
not  meet  the  90  percent  availability  criterion  as  stated  in 
the  DTEMP. 


•  The  2D  EEMT  units  assigned  to  the  Primary  Power 
laboratory  were  available  for  training  90.8  percent  of  the 
required  time. 

•  The  2D  EEMT  units  allocated  to  the  Primary  Power  extra 
study/ remediation  laboratory  were  available  for  training  46 
percent  of  the  time. 


Maintainability 


This  element  of  the  TCT  was  defined  els  the  ability  of  the  ET  "A"  School 
Support  Division  to  provide  on-site  maintenance  support  subsequent  to  contractor 
training  and  using  existing  spares  packages.  The  information  required  to  evaluate 
the  maintainability  issue  was  collected  from  ET  "A"  School  Support  Division  Daily 
Production  Reports,  Down  Equipment  Daily  Logs,  interviews  with  Support  Division 
personnel  and  review  of  maintenance  contract  commitments  and  plans. 

The  extent  to  which  the  Support  Division  personnel  were  able  to 
complete  the  118  corrective  actions  necessary  to  maintain  and  logistically  support 
the  2D  EEMT  equipment  is  related  to  two  factors:  extent  of  maintenance  training 
and  adequacy  of  spares  packages. 


Training 

Contractor  specifications  for  maintenance  training  were  identified  as  4 
weeks  of  formal  training  plus  on-the-job  training  (Pine,  Daniels,  A:  Malec,  1981). 
Interviews  with  ET  "A”  School  Support  Division  personnel  revealed  that  no  formal 
training  program  existed  for  2D  EEMT  maintenance  training  during  the  TCT. 
Although  a  training  workshop  was  conducted  by  the  contractor  in  November  1982, 
the  workshop  emphasized  preventive  maintenance  and  was  attended  by  both 
maintenance  and  nonmaintenance  personnel.  Some  ET  "A"  School  maintenance 
personnel  who  attended  the  workshop  were  no  longer  at  ET  "A"  School  by  the  time 
of  the  TCT.  Therefore,  the  Support  Division  personnel  who  were  responsible  for 
2D  EEMT  maintenance  during  the  TCT  received  only  on-the-job  training.  Since 
extensive  experience  and  training  in  general  electronics  repair  is  endemic  to  ET 
"A"  School,  Support  Division  personnel  had  the  skill  and  confidence  to  undertake 
limited  in-house  maintenance  of  the  2D  EEMT  systems  and  components.  For 
example,  faulty  boards  in  the  microprocessor  were  usually  repaired  in  the  Support 
Division  workshop.  CRT  actions  were  limited  to  replacement  or  retaping  of  touch 


panels  and  calibration.  Videodisc  player  maintenance  was  confined  to  swapping 
units.  Actual  repairs  were  performed  by  authorized  commercial  shops  off-base  or 
by  contractor  personnel. 


Adequacy  of  Spares  Packages 

In  general  the  spares  inventory  available  during  the  TCT  was  not 
adequate  to  meet  equipment  maintenance  requirements.  This  was  due  in  part  to 
the  need  to  gain  experience  with  equipment  malfunction  patterns  before  the 
optimal  spare  parts  package  could  be  determined.  As  previously  indicated  the  18 
2D  EEMT  units  assigned  to  the  Primary  Power  course  were  available  for  training 
72  percent  of  the  time.  Of  the  eight  units  assigned  to  the  Primary  Power 
laboratory,  much  of  the  downtime  was  accounted  for  by  only  one  unit  (serial 
number  5696/11)  which  was  inoperable  for  approximately  50  percent  of  the 
duration  of  the  TCT.  These  same  records  indicated  that  the  10  units  assigned  to 
the  Primary  Power  extra  study/remediation  laboratory  were  available  for  46 
percent  of  the  TCT  duration  overall,  with  two  units  (serial  numbers  6241/25  and 
6015/35)  inoperable  100  percent  of  the  time,  and  7  of  the  10  units  inoperable  at 
the  end  of  the  TCT  period.  In  almost  all  cases,  inoperability  was  due  to  a  lack  of 
spare  parts  rather  than  a  lack  of  staff  capability  to  effect  repairs. 

Although  spare  components  for  each  of  the  2D  EEMT  subsystems  were 
provided  by  NPRDC  prior  to  the  TCT,  and  contractor  delivered  spare  components 
were  subsequently  provided,  support  personnel  ordered  parts  as  needed  to  effect 
immediate  repairs.  The  apparent  inadequacy  of  the  spare  parts  inventory  can  be 
attributed  largely  to  the  high  failure  rate  of  12"  touch  panels.  Spare  panels  were 
difficult  to  obtain  and,  therefore,  were  always  back  ordered.  During  the  TCT,  an 
improved  version  of  the  12"  touch  panel  was  introduced  to  replace  the  original 
touch  panels.  This  change,  while  expected  to  reduce  the  number  of  touch  panel 
failures  in  the  future,  commonly  resulted  in  delays  of  3  to  6  months  in  parts 
availability.  A  "normal”  spares  package  configuration  based  on  actual  classroom 
performance  needs  was  being  reviewed  by  Navy  personnel  at  the  conclusion  of  the 
TCT  and  alternative  configurations  of  and  sources  for  touch  panels  were  being 
considered. 

It  should  be  noted  that  the  ability  of  Support  Division  personnel  to 
maintain  the  Primary  Power  2D  EEMT  units  could  have  been  worse  except  for  a 
pool  of  "spare"  units  which  was  made  available  because  of  the  TCT.  The  20  2D 
EEMT  units  assigned  to  the  Communications  training  area  were  taken  out  of 
service  at  DRI's  request  to  prevent  student  contact  with  the  equipment  prior  to 
Primary  Power  training.  These  "spare"  units  served  as  sources  of  parts  during  the 
TCT. 


Summary 


Results  of  the  analysis  of  equipment  maintainability  showed  that: 

•  ET  "A"  School  personnel  alone  were  not  able  to  adequately 
maintain  and  logistically  support  the  2D  EEMT  during  the 
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TCT.  A  combination  of  limited  training  and  lack  of  needed 
spare  parts  resulted  in  much  of  the  maintenance  work 
being  conducted  by  contract  personnel  or  through 
agreements  with  component  manufacturers. 

•  The  maintainability  of  the  2D  EEMT  system  was 
substantially  affected  by  touch  panel  malfunctions.  These 
panels  were  difficult  to  obtain  and  were  replaced  with  a 
newer  version  which  often  resulted  in  delays  of  3  to  6 
months. 

•  The  development  of  a  more  comprehensive  spare  parts 
package  based  on  experience  with  the  prototype  units  and 
the  use  of  a  maintenance  contract  will  substantially 
improve  the  maintainability  of  the  2D  EEMT  systems. 

•  The  maintenance  role  of  Support  Division  personnel  must 
be  more  clearly  defined  and  appropriate  training  provided 
accordingly.  The  fact  that  Support  Division  personnel  have 
extensive  backgrounds  in  electronics  does  not  diminish  the 
need  for  a  formal  training  program. 


>ropriateness  to  Primary  Power  Training 


Two  important  considerations  are  the  extent  to  which  a  trainer  meets 
course  training  objectives  and  the  ease  with  which  it  can  be  accommodated  into 
the  training  environment.  Appropriateness  for  training  objectives  was  evaluated 
by  examining  the  lessonware  format  and  content,  considering  safety  issues,  and 
observing  student-instructor  ratios.  Accommodation  to  the  environment  was 
evaluated  by  examination  of  the  physical  requirements  of  operating  the  2D  EEMT 
system  and  the  transportability  of  the  units  from  one  location  to  another.  A 
comparison  of  the  AN/SPS-10  and  the  2D  EEMT  on  these  issues  is  provided  in 
Table  8.  The  information  necessary  to  evaluate  these  issues  was  obtained  from 
structured  interviews  with  several  ET  "A"  School  persons  and  with  Instructional 
Program  Development  (IPD)  Center  persons  familiar  with  the  Primary  Power 
training  objectives,  and  review  of  2D  EEMT  documentation. 


Training  Lessons 

Format.  According  to  documentation  and  instructors  who  were 
interviewed,  the  free-play  lessonware,  which  was  developed  by  the  manufacturer, 
complements  and  encourages  the  use  of  technical  manuals,  allows  students  access 
to  areas  of  the  radar  equipment  which  are  off  limits  for  training  on  the 
operational  equipment  (due  to  safety  considerations),  and  trains  logical  and 
technical  skills  well. 

Structured  lessonware,  also  developed  by  the  manufacturer  for  Primary 
Power  training,  was  not  used  during  the  period  of  the  TCT.  The  structured 
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Table  8 


Appropriateness  of  the  2D  EEMT  to 
Primary  Power  Training  as  Compared  to  the  AN/SPS-10 


Category 

AN/SPS-10 

2D  EEMT 

Training  Lessons: 

Format 

Free-play;  problem 
solving 

Free-play;  problem 
solving 

Structured3 

Content 

Primarily  wire  breaks; 
meets  training 
requirements 

Primarily  component 
failures;  meets  training 
requirements 

Number  of  Lessons 

Total  Primary  Power 
laboratory  =  25^; 

In  use  per  AN/SPS-10 

7 

Total  Primary  Power 
laboratory  =  29;  available 
to  each  2D  EEMT  =  29 

Safety: 

Operation 

Very  poor 

Very  good 

Training 

Good0 

Poord 

Student/Instructor  Ratio 

Approximately  15:1 
with  laboratory 
assistant;  primarily 
team  training 

Approximately  10:1; 
primarily  individualized 
training6 

Physical  compatibility: 

Power  requirements 

120  VAC1 

120  VAC 

Air  conditioning 
requirements 

Met  by  existing 
capacity 

Met  by  existing 
capacity 

Transportability 

Very  poor 

Very  good 

aTutorial  lessonware  was  not  in  use  during  the  TCT. 

b Additional  hand-inserted  problems  on  an  as-needed  basis  can  increase 
this  pool  to  approximately  60. 

cRegulur  use  is  seen  to  increase  student  respect  for  operating  dangerous 
equipment. 

dThere  are  no  feedback  messages  for  safety  errors;  there  is  no  instructor 
monitoring  for  safety  errors;  the  system  allows  unsafe  procedures  to  be 
performed. 

eOnly  reflects  TCT  period. 

*  Power  is  stepped  up  internally  in  the  equipment  to  20,000  volts. 


lessonware  was  lengthy,  inflexible  in  application,  and  was  never  modified  to 
reflect  changes  in  the  picture  base.  Specifically,  each  structured  lesson  could 
require  hours  to  complete,  whereas  free-play  lessons  could  be  completed  in 
minutes.  In  addition,  the  structured  lessonware  was  linear  in  format;  a  student 
was  required  to  complete  the  entire  lesson  once  he  or  she  began  the  lesson.  There 
were  very  few  students  who  required  training  or  additional  work  in  an  entire 
subject  area;  rather,  students  usually  required  extra  work  on  specific  elements 
which  were  addressed  more  appropriately  by  free-play  lessonware.  Finally,  as 
modifications  to  the  picture  base  were  made,  the  capability  to  make 
modifications  to  the  structured  lessonware  to  reflect  those  changes  did  not  exist; 
therefore,  the  structured  lessonware  was  quickly  out  of  date. 

Content.  Appropriateness  to  vacuum  tube,  solid  state,  and  LSI 
technology  training  was  determined  to  be  relevant  for  Primary  Power  training 
only  with  respect  to  vacuum  tube  technology  training.  The  free-play  lessonware 
was  seen  to  be  effective  in  meeting  vacuum  tube  training  needs.  Solid  state  and 
LSI  technology  training  is  appropriate  to  other  areas  of  AN/SPS-10  training.  The 
appropriateness  of  2D  EEMT  lessonware  to  these  training  needs  was  not  examined 
during  the  TCT.  The  free-play  lessons  emphasize  component  repair;  this  emphasis 
complements  operational  equipment  lessons  which  emphasize  wire  repairs. 


Safety 


Several  comments  were  made  regarding  the  issue  of  safety  training.  The 
person-equipment  interface  was  seen  to  be  much  safer  with  the  2D  EEMT  system 
than  with  the  operational  AN/SPS-10  trainers.  Students  were  encouraged  in  the 
2D  EEMT  laboratory  to  use  the  eraser  end  of  a  pencil  to  touch  the  screens  to 
ensure  adherence  to  basic  safety  procedures,  i.e.,  keeping  hands  out  of  high 
voltage  areas.  Several  criticisms  were  leveled,  however,  regarding  the  extent  to 
which  the  lessonware  addressed  basic  safety  principles. 

The  ET  "A"  School  has  a  significant  safety  problem  because  of  the  high 
voltages  in  use  with  the  Primary  Power  radar  equipment.  For  this  reason,  certain 
parts  of  the  Primary  Power  radar  equipment  are  off  limits  to  students  during 
laboratory  training.  One  advantage  to  the  2D  EEMT  lessonware  is  that  it  permits 
student  access  to  these  areas  through  the  picture  base,  allowing  a  more 
comprehensive  understanding  of  the  equipment.  Unfortunately,  the  lessonware 
also  permits  students  to  take  in-line  voltage  measurements  and  perform  other 
actions  which  are  not  in  compliance  with  safe  operating  procedures.  In  addition, 
there  is  no  aural  or  visual  feedback  to  the  student  or  the  instructor  when  a  safety 
violation  has  occurred.  One  laboratory  instructor  indicated  that  although  it  was 
possible  to  restrict  student  access  to  specific  areas  of  the  picture  base,  the 
required  programming  was  too  complex  to  undertake.  The  individual  training 
format  of  the  2D  EEMT  may  also  have  contributed  to  the  problem  since  the  team 
approach  used  with  operational  AN/SPS-10  equipment  encourages  one  student  to 
monitor  the  activities  of  the  other  student  and  help  prevent  violations  of  the 
safety  rules.  Finally,  students  who  commit  safety  violations  in  the  laboratory  are 
required  to  serve  as  safety  monitors  during  other  laboratory  training  shifts.  These 
students  act  as  additional  protection  against  safety  violations  in  the  laboratories. 
These  students  were  not  assigned  to  perform  this  function  for  the  2D  EEMT 
equipment  during  the  TCT  period. 
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Student-Instructor  Ratio 


The  student-instructor  ratio  was  not  manipulated  during  the  TCT.  It  did 
fluctuate,  however,  during  the  TCT  period.  The  student-instructor  ratio  was 
increased  due  to  increased  class  size  to  reduce  a  training  backlog.  However, 
separate  instructors  for  operational  laboratory  and  2D  EEMT  laboratory  training 
lowered  the  student-instructor  ratio.  Despite  these  fluctuations,  the  use  of  2D 
EEMT  units  has  the  potential  to  increase  the  student-instructor  ratio  for  the 
following  reasons: 

1.  Operational  equipment  laboratory  instructors  were  often 
unfamiliar  with  operation  of  the  2D  EEMT.  Once  ET  "A” 

School  instructors  become  familiar  with  2D  EEMT 
operation,  the  need  for  a  separate  2D  EEMT  laboratory 
instructor  could  end. 

2.  2D  EEMT  touch  panel  problems  and  system  lockup 
occurrences  necessitated  the  presence  of  a  2D  EEMT 
knowledgeable  person  to  effect  immediate  repairs  when 
possible.  Comments  received  from  instructors  and 
students  indicated  that  the  absence  of  2D  EEMT 
knowledgeable  persons  in  extra/study  remediation 
laboratories  often  resulted  in  2D  EEMT  units  being  taken 
out  of  use  for  minor  problems.  Correction  of  problems 
associated  with  touch  panels  and  system  lockup  would 
reduce  the  need  for  separate  2D  EEMT  laboratory 
instructors. 

3.  The  reduced  risk  of  injury  with  2D  EEMT  use  reduces  the 
need  for  performance  monitoring  to  the  degree  required 
in  the  operational  equipment  laboratory.  Imple¬ 
mentation  of  oral  or  visual  feedback  or  use  of  student 
safety  monitors  could  address  adherence  to  safe 
operating  procedures  without  adding  additional 
instructor  responsibility. 

4.  The  2D  EEMT  units  in  Primary  Power  laboratory  training 
were  used  only  during  3  of  the  U  days  of  laboratory 
training  and  were  not  fully  utilized  then.  More  efficient 
use  of  the  2D  EEMT  (increased  student  contact  time) 
could  allow  an  increased  student/instructor  ratio. 

5.  Current  Primary  Power  laboratory  configurations  require 
separate,  although  adjacent,  interconnected  rooms  for 
AN/SPS-10  and  2D  EEMT  equipment.  This  arrangement 
necessitates  two  instructors.  Rearrangement  to  permit 
visual  monitoring  by  one  instructor  could  increase  the 
student-instructor  ratio. 


No  formal  evaluation  of  the  student-instructor  ratio  issue  was  undertaken  during 
the  TCT,  nor  were  any  of  these  recommendations  implemented.  During  the  TCT, 


the  Primary  Power  2D  EEMT  laboratory  student/ instructor  ratio  was  usually  8:1 
since  there  were  eight  2D  EEMT  units  in  that  laboratory.  Except  for  TCT- 
required  team  training,  most  students  worked  individually.  The  operational 
equipment  laboratory  student/instructor  ratio  ranged  from  15-20:1  depending  on 
the  presence  of  an  assistant  and/or  the  classroom  instructor.  The  laboratory 
instructor  was  primarily  responsible  for  training  and  student  performance 
monitoring. 


Compatibility  to  the  Primary  Power  Environment 

The  2D  EEMT  equipment  was  considered  to  be  compatible  with  the 
Primary  Power  laboratory  training  environment.  The  2D  EEMT  units  were 
initially  housed  in  existing  laboratory  space  as  they  were  acquired.  A  portion  of 
an  existing  electronic  equipment  testing  area  (approximately  200  square  feet)  was 
later  slightly  modified  to  house  the  Primary  Power  laboratory  2D  EEMT  units.  No 
electrical  or  air  conditioning  modifications  were  required. 

The  2D  EEMT  was  also  compatible  with  the  extra  study/remediation 
laboratory.  Ten  2D  EEMT  units  were  placed  along  the  perimeter  of  the  two  rooms 
used  for  Primary  Power  extra  study/remediation.  No  electrical  or  air 
conditioning  modifications  were  made  to  the  rooms  to  accommodate  the  units. 
Students  operating  the  2D  EEMT  units,  individually  or  in  teams,  required  little 
direct  interaction  with  extra  study/remediation  instructors. 


T  ransportability 

The  issue  of  assembly  and  disassembly  as  they  relate  to  transportability 
of  the  units  was  only  peripherally  addressed.  Since  several  additional  2D  EEMT 
systems  were  purchased  and  implemented  in  other  training  areas  after  the  initial 
implementation  in  the  Primary  Power  laboratory,  the  need  to  transport  the  2D 
EEMT  systems  to  various  locations  did  not  arise.  Since  the  2D  EEMT  units  were 
mounted  on  casters,  transportability  would  not  have  been  a  significant  problem. 
The  assembly/disassembly  issue  was  indirectly  addressed  by  the  maintenance 
activities  of  the  Support  Division.  Components  of  the  2D  EEMT  systems  were 
removed  to  the  Support  Division  shop  area  or  swapped  among  systems  to  maintain 
training  availability  throughout  the  TCT  period.  These  actions  involved 
movement  of  components  among  several  different  rooms  located  in  two  separate 
buildings.  No  significant  problems  were  encountered. 


Interoperability  with  the  Navy  CMI  System 

The  2D  EEMT  systems  were  not  directly  tied  into  the  Navy  CMI  system 
during  the  TCT.  Primary  Power  laboratory  and  extra  study/remediation 
laboratory  student  performance  was  recorded  on  floppy  disks  assigned  to  the 
students  by  the  2D  EEMT  lab  instructor.  Through  the  use  of  printers  located  at 
instructor  consoles  in  the  2D  EEMT  laboratory,  printouts  of  student  performance 
were  obtained  and  placed  in  student  files.  The  potential  of  interfacing  the  2D 
EEMT  performance  tracking  system  with  the  Navy  CMI  system  was  not  examined 
during  the  TCT. 


Sum  mar 


Analysis  of  the  elements  related  to  the  appropriateness  of  the  2D  EEMT 
to  the  Primary  Power  training  program  revealed  that: 

•  The  2D  EEMT  free-play  lessonware  is  appropriate  to 
meeting  the  Primary  Power  training  objectives. 

•  The  tutorial  or  structured  lessonware  is  too  lengthy  and 
inflexible,  and  has  not  been  modified  to  reflect  changes  in 
the  videodisc  image  base. 

•  The  2D  EEMT  system  is  safer  to  operate  than  AN/SPS-10 
equipment  although  there  are  several  weaknesses  in  the 
lessonware  related  to  the  training  of  safe  operating 
procedures.  These  include  the  capability  to  take  voltage 
measurements  in  a  manner  which  does  not  reflect  safe 
operating  procedures  and  the  lack  of  visual  or  aural 
feedback  for  safety  violations. 

•  The  2D  EEMT  system  is  compatible  with  the  existing 
Primary  Power  laboratory  facilities  and  is  easily 
transportable  through  disassembly  into  component  parts. 

•  The  2D  EEMT  is  easily  assembled  and  disassembled  and  is 
highly  transportable. 

•  The  data  management  features  of  the  2D  EEMT  were  not 
examined  with  regard  to  the  Navy  CMI  system.  The 
features  which  existed  during  the  TCT  were  very  limited. 


Adaptability 


The  scenarios  discussed  in  this  section  describe  situations  in  which  the 
2D  EEMT  could  be  used  in  situations  other  than  the  current  use  in  the  Primary 
Power  laboratory  and  in  extra  study/remediation.  The  capability  of  the  2D  EEMT 
to  adapt  to  various  changes  was  considered,  and  conclusions  were  drawn  based  on 
observation  and  interviews  with  ET  "A"  School  and  IPD  Center  personnel. 


Scenario  1:  Changes  in  Representative  Operational  Equipment 

The  Navy  has  considered  the  replacement  of  the  AN/SPS-10  with  the 
AN/5PS-67  in  its  materials  inventory.  For  the  Primary  Power  laboratory,  this 
would  represent  a  change  in  the  representative  operational  equipment.  Using  this 
potential  change  as  the  basis  for  the  scenario,  the  ability  of  the  2D  EEMT  to  be 
adapted  to  this  change  was  examined. 


The  hardware  and  software  of  the  2D  EEMT  system  would  not  require 
modification.  Changes  to  the  lessonware  and  the  videodisc  image  base  would  be 
determined  by  the  extent  of  the  differences  between  the  equipments  (AN/SPS-10 
and  AN/SPS-67)  and  the  costs  associated  with  such  changes.  Several  ET  "A" 
School  persons  are  currently  competent  to  author  lessons  on  the  2D  EEMT  system. 
Estimates  of  3  to  5  weeks  of  training  were  given  to  attain  authoring  competency, 
which  would  not  be  a  significant  barrier  to  developing  new  lessonware  as  needed. 

Changes  in  the  representative  operational  equipment  could  be  costly, 
however,  since  the  cost  of  incorporating  changes  to  the  videodisc  may  be  the 
same  as  developing  a  new  disc.  Modifications  to  the  videodisc  may  be  limited  to 
adding  frames  to  the  videotape  master  and  remastering  the  disc  or  may  extend  to 
developing  a  new  image  base  entirely.  The  capability  to  develop  a  new  videodisc 
image  base  is  not  currently  available  at  Great  Lakes  Naval  T raining  Station.  The 
cost  of  having  a  new  image  base  developed  and  a  master  disk  prepared  could  be 
extensive. 


Scenario  2:  Changes  in  the  Instructional  Environment 

Use  of  the  2D  EEMT  in  Primary  Power  laboratory  and  extra 
study/remediation  has  been  demonstrated.  Use  of  the  2D  EEMT  in  classroom  for 
demonstration  and  structured  training  is  considered  in  this  scenario. 

Although  structured  lessonware  was  developed  by  the  contractor  for  the 
initial  2D  EEMT  implementation,  it  was  not  used  in  training  during  the  TCT 
because  of  its  length  (approximately  8  hours)  and  the  fact  that  the  lessons  were 
not  revised  to  reflect  changes  in  the  videodisc  image  base.  Use  of  structured 
lessonware  in  the  classroom  could  assist  students  in  gaining  familiarity  with  the 
operational  equipment  and  enhance  blackboard  training  procedures.  Adaptation  of 
the  2D  EEMT  to  classroom  training  would  not  require  changes  to  the  hardware, 
software,  or  videodisc  image  base.  It  would  require  redesign  of  the  structured 
lessonware  to  synchronize  it  with  the  videodisc  image  base.  These  capabilities 
exist  within  the  ET  "A"  School. 


Scenario  3:  Changes  in  Personnel  Skill  Level 

The  2D  EEMT  has  been  demonstrated  at  the  ET  "A”  School  training  level. 
Its  use  could  be  expanded  to  less  skilled  areas  (e.g.,  BEicE)  or  to  advanced  training 
areas  (e.g.,  "C"  School).  These  two  applications  are  considered  in  this  scenario. 

Less  skilled  training.  Application  to  a  less  skilled  area  would  not 
necessarily  require  hardware  or  software  changes  to  the  2D  EEMT.  Lessonware 
and  videodisc  image  base  changes  may  be  required  since  the  basic  electronics 
training  would  require  generic  component  images  and  problems.  If  the  AN/SPS-10 
equipment  could  be  considered  generic  for  basic  electronics  training,  some  of  the 
existing  image  base  may  be  appropriate  to  basic  electronics  training.  Appropriate 
lessonware  would  need  to  be  designed  for  basic  electronics.  Because  the  basic 
electronics  training  format  is  currently  self-paced,  the  individualized  training 
format  of  the  2D  EEMT  is  easily  adaptable  to  this  environment. 
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Advanced  training.  Application  to  advanced  training  areas  would  not 
require  changes  to  the  hardware  and  software.  Since  the  AN/SPS-10  is  not  taught 
at  the  "C"  School  level,  changes  to  the  lessonware  and  videodisc  image  base  would 
be  dependent  on:  (1)  the  level  of  difficulty  of  existing  lessons  and  (2)  the 
appropriateness  of  the  videodisc  image  base  in  representing  various  levels  of 
detail  of  other  operational  equipment. 


Scenario  4:  Changes  in  the  Rwsical  Training  Environment 


On-the-job  training  is  a  significant  part  of  Navy  electronics  training. 
The  2D  EEMT  could  be  used  to  provide  and  monitor  OJT  or  serve  as  a 
performance  aid  to  shipboard  technicians.  These  applications  are  considered  in 
this  scenario. 

Q3T.  The  flexibility  of  the  2D  EEMT  system,  as  illustrated  in  the 
previous  scenarios,  suggests  that  application  to  OJT  in  both  dock-side  and 
shipboard  environments  may  be  feasible.  The  performance  monitoring  capability 
of  the  2D  EEMT,  while  not  specifically  studied  during  the  TCT,  may  be 
compatible  with  the  Personnel  Qualification  System. 

However,  the  2D  EEMT  has  been  demonstrated  only  in  a  formal  training 
environment,  i.e.,  in  established  classroom  buildings  with  in-residence  support 
personnel  and  access  to  spares  through  an  established  system.  The  application  of 
the  2D  EEMT  in  dock-side  training  centers  or  on  ships  must  be  accompanied  by 
the  development  of  an  adequate  support  system. 

Performance  aid.  The  objectives  of  performance  aiding  are  in  contrast 
to  those  for  training.  While  training  requires  teaching  problem  solving  skills 
through  practice,  performance  aiding  requires  technical  information  storage  and 
retrieval  and  problem  solving  guidance  for  unspecified  malfunctions.  For  these 
reasons,  the  software  and  lessonware  of  the  2D  EEMT  may  not  be  easily  adapted 
to  meet  the  objectives  of  performance  aiding.  Revisions  to  the  software  would 
probably  require  costly  modifications.  The  hardware  and  image  base  may  not 
require  extensive  modifications,  assuming  the  salt  water  environment  and  the 
completeness  of  the  image  base  in  representing  the  operational  equipment  are  not 
significant  problems. 


Summary 

Analysis  of  the  adaptability  of  the  2D  EEMT  to  an  alternative  training 
environment  suggests  that: 

•  The  2D  EEMT  is  highly  adaptable  to  changes  in  the 
representative  operational  equipment.  The  primary  2D 
EEMT  modifications  required  would  be  to  the  lessonware 
and  videodisc  image  base. 

•  The  2D  EEMT  is  highly  adaptable  to  introduction  into  the 
Primary  Power  classroom.  The  major  system  modification 


necessary  would  be  development  of  appropriate  structured 
lessonware. 


•  The  2D  EEMT  is  adaptable  to  changes  in  personnel  skill 
level.  Modifications  to  lessonware  and  videodisc  image 
base  would  be  required  to  match  the  skill  level  in  question. 

•  While  the  training  approach  employed  by  the  2D  EEMT  may 
be  adaptable  to  OJT  environments,  and  for  performance 
aiding  in  particular,  the  current  configuration  of  the  2D 
EEMT  would  require  extensive  changes.  Therefore,  the  2D 
EEMT  is  not  considered  to  be  easily  adaptable  to  dock-side 
or  shipboard  OJT  environments. 


User  Acceptance 


The  importance  of  user  acceptance  in  the  assessment  of  training 
effectiveness  of  simulators  is  well  demonstrated.  This  study  was  designed  to 
assess  the  extent  of  user  acceptance  for  the  2D  EEMT  system  within  the  ET 
School  environment.  In  particular,  this  study  addressed  the  following  hypotheses: 

•  Instructors  accept  the  2D  EEMT  system  as  a  viable  means 
of  providing  ET  School  laboratory  and  remedial  training. 

•  Students  accept  the  2D  EEMT  system  as  a  viable  means  of 
providing  ET  School  laboratory  and  remedial  training. 

In  addition  to  obtaining  user  acceptance  data,  this  study  provided  other 
supporting  information  necessary  to  address  some  of  the  training  capabilities 
evaluation  questions  outlined  in  the  previous  studies,  especially  concerning 
person-equipment  interface,  system  compatibility  with  the  school  environment, 
hands-on  training,  and  appropriateness  of  training. 


Method 


The  subjects  for  this  study  included  all  students  and  instructors  that  had 
experience  with  the  2D  EEMT  trainer  during  the  course  of  the  TCT.  Laboratory, 
classroom,  and  remediation  instructors  were  asked  to  complete  a  written 
questionnaire  (see  Section  A-4  in  Appendix  A)  at  the  close  of  the  evaluation 
period.  Additional  on-site  structured  interviews  were  conducted  by  DRI  staff 
with  key  ET  School  personnel  (Section  A-5  in  Appendix  A  presents  the  interview 
guide).  Written  critiques  were  also  administered  by  ET  School  personnel  to  all 
students  participating  in  the  TCT  at  the  end  of  Primary  Power  training  (rf. 
Section  A-2  in  Appendix  A). 
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Instructor  Critique  Results 

Instructor  critiques  were  completed  by  48  instructors  and  training 
supervisors  assigned  to  the  radar  branch  of  ET  School.  Thirty-two  respondents  1 

reported  using  the  2D  EEMT  equipment  as  a  regular  part  of  radar  training  and  all  | 

respondents  reported  having  at  least  limited  EEMT  experience.  j 

Based  on  their  experiences  with  the  EEMT,  respondents  were  asked  to  ■ 

use  a  5-point  scale  to  rate  their  agreement  with  statements  about  simulator  i 

training  in  general.  Mean  ratings  were  calculated  and  are  shown  in  parentheses  in 
the  sections  that  follow.  Two-tailed  t- tests  were  then  computed  for  each 
statement  to  identify  those  mean  ratings  that  differed  significantly^  from  a 
noncommittal  rating  (i.e.,  3.00).  Of  the  10  general  statements  included,  only  one 
drew  moderate  support:  that  including  simulators  in  training  is  a  good  idea  (3.66). 

Instructors  disagreed  with  6  of  the  10  statements:  that  simulators  can  be  more 
effective  than  actual  equipment  (1.72),  can  provide  equivalent  training  with 
actual  equipment  (2.20),  can  provide  adequate  training  at  a  cost  savings  (2.32),  are 
more  reliable  than  actual  equipment  (1.87),  teach  safety  training  better  than 
actual  equipment  (1.30),  and  provide  more  variety  of  training  than  actual 
equipment  (2.43). 

When  asked  to  rate  the  importance  of  13  factors  which  are  considered 
relevant  to  simulator  design,  respondents  agreed  that  all  were  relatively 
important.  The  mean  ratings  obtained  were  greater  than  3.00  in  every  case.  The 
most  important  factors  were  identified  as  those  with  mean  ratings  significantly 
greater  than  4.00,  again  using  the  t-test  approach.  Four  factors  met  this 
criterion:  the  ability  to  simulate  complexity  of  actual  problems  encountered  in 
the  field  (4.30),  the  capability  of  Navy  personnel  to  modify  existing  lessons  or 
create  new  lessons  for  the  training  simulator  (4.49),  the  reliability  of  performance 
of  the  equipment  (4.68),  and  the  ease  of  maintenance  of  the  equipment  (4.63). 

Respondents  also  rated  their  agreement  with  statements  that  focused 
specifically  on  the  free-play  troubleshooting  lessonware,  the  authoring  software, 
and  the  data  management  aspects  of  the  2D  EEMT  system.  Mean  ratings  for  6  of 
the  10  statements  were  noncommittal  (i.e.,  did  not  differ  significantly  from  3.00); 
the  remaining  four  statements  were  not  supported.  Instructors  disagreed  with 
statements  that:  the  EEMT  system  management  does  not  add  significantly  to  the 
workload  (2.42),  student  data  produced  by  the  EEMT  system  are  helpful  in  student 
management  (2.42),  student  data  produced  by  the  EEMT  system  are  useful  in 
assessing  student  abilities  (2.36),  and  the  authoring  software  provides  an  effective 
method  of  developing  new  lessons  (2.39).  It  should  be  noted,  however,  that  only 
five  of  the  respondents  indicated  having  hands-on  experience  with  the  authoring 
software.  Because  tutorial  lessons  were  not  used  by  ET  School  personnel  during 
the  TCT  period,  critique  data  related  to  those  lessons  were  omitted  from  analysis. 

Complete  results  from  the  instructor  critiques  are  provided  in  Table  B-5  of 
Appendix  B. 


.05  level  of  significance  was  selected  for  each  comparison. 
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In  addition  to  critique  ratings,  34  of  the  48  respondents  provided  written 
comments  on  various  aspects  of  the  2D  EEMT  system.  Eleven  instructors  believed 
there  was  a  general  need  to  improve  system  reliability  and  nine  more  noted 
specific  reliability  problems  (e.g.,  touch  panel  drift,  system  lockup)  that 
interfered  with  training.  Another  major  concern  was  the  inability  of  the  system 
to  provide  appropriate  safety  training  and/or  monitor  safety-related  student 
errors.  Additional  comments  focused  on  the  need  for  better  representations  of 
test  equipment  use,  improved  data  management,  and  expansion  of  the 
photographic  data  base.  A  frequent  remark  was  that  the  EEMT  is  good  for 
remedial/extra  study  applications  but  it  cannot  take  the  place  of  the  actual 
equipment  for  laboratory  training. 

The  results  of  the  Instructor  Critiques  were  substantiated  by  additional 
information  collected  during  informal  on-site  interviews  with  ET  School 
personnel.  Overall,  instructors  thought  that  the  2D  EEMT  could  be  a  useful 
training  supplement  to  the  actual  AN/SPS-IO  equipment  but  there  were 
reservations  about  the  current  prototype.  Their  principal  concerns  can  be 
summarized  as  follows: 

•  The  system  needs  improvement  in  reliability  and 
maintainability. 

•  The  system  should  incorporate  a  means  of  training  and 
monitoring  safety  procedures. 

•  Data  management  features  need  improvement  for  easier 
access  to  student  records  and  greater  data  interpretability. 

•  The  free-play  lesson  ware  should  be  improved  to  provide 
more  assistance  to  the  student  and  expanded  to  include  the 
full  range  of  troubleshooting  problems. 


Student  Critique  Results 

Student  critiques  were  completed  by  109  students  following  their 
AN/SPS-10  Primary  Power  laboratory  training.  This  sample  included  49  students 
who  were  assigned  to  High  EEMT  conditions  and  48  students  who  were  assigned  to 
Low  EEMT  conditions.  Only  12  students  assigned  to  the  Operational  Only  training 
mode  completed  critiques. 

Respondents  were  asked  to  use  a  5-point  scale  to  rate  their  agreement 
with  statements  about  the  operational  equipment  training  and,  if  applicable,  about 
the  2D  EEMT  training  they  received.  Mean  ratings  were  computed  and  are  shown 
in  parentheses.  A  complete  summary  of  these  student  critique  ratings  are 
provided  in  Table  B-6  of  Appendix  B.  Comments  about  specific  training  likes  and 
dislikes  were  also  collected. 

The  student  critique  results  were  examined  using  three  different 
approaches.  First,  and  most  important,  a  descriptive  analysis  of  student  opinions 
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was  performed.  The  particular  focus  of  this  analysis  was  2D  EEMT  training. 
Second,  an  analysis  of  variance  approach  was  used  to  examine  whether  student 
opinions  differed  as  a  function  of  training  conditions.  Finally,  for  comparable 
critique  items,  matched  sample  t-tests  were  conducted  to  compare  operational 
equipment  and  2D  EEMT  training. 

Operational  equipment  training.  Student  opinions  were  consistently 
favorable  regarding  operational  AN/SPS-10  training.  Students  strongly  disagreed 
with  statements  that  training  time  on  the  equipment  was  often  wasted  (1.65)  and 
that  equipment  breakdowns  interfered  with  training  (1.54);  they  most  strongly 
agreed  with  statements  that  they  feel  comfortable  working  on  the  equipment 
(4.09)  and  that  a  variety  of  training  problems  were  provided  (3.97). 

Of  the  eight  statements  concerning  operational  equipment  training 
included  in  the  critiques,  students  disagreed  on  only  two  as  a  function  of  training 
conditions.  Student  ratings  of  the  statement  that  there  was  enough  training  time 
on  the  equipment  differed  significantly  among  High  EEMT  (2.76),  Low  EEMT 
(3.38),  and  Operational  Only  (3.91)  conditions  [F(2,98)  =  7.31,  £<.00l] .  These 
groups  also  differed  with  respect  to  a  similar  statement  that  more  time  should  be 
spent  on  troubleshooting  [F(2,97)  -  3.39,  j>  <.05];  mean  ratings  were  3.85  for  High 
EEMT,  4.00  for  Low  EEMT,  and  3.00  for  Operational  Only  conditions.  Thus,  while 
students  in  all  three  conditions  thought  that  more  hands-on  troubleshooting  time 
was  necessary,  students  in  EEMT  training  groups  expressed  the  strongest  need. 

These  findings  were  further  supported  by  written  comments.  Seventeen 
EEMT-trained  students  indicated  that  more  troubleshooting  or  hands-on  training 
on  the  AN/SPS-10  equipment  was  needed  and  !  1  noted  that  operational  equipment 
provides  the  most  appropriate  training  format.  Additional  written  comments 
expressed  the  importance  of  operational  equipment  training  for  matters  of  safety, 
the  desire  to  see  more  "nonstandard  faults,"  the  belief  that  paired  training  could 
be  safer  or  more  efficient  than  individual  training,  and  the  opinion  that  working 
individually  is  more  effective  than  training  in  pairs. 

2D  EEMT  training.  Overall,  student  responses  to  the  1 1  statements  that 
focused  on  EEMT  training  were  either  noncommittal  or  marginally  favorable. 
Mean  ratings  of  agreement  ranged  from  a  low  of  2.43  to  a  high  of  3.53  (rf.  Table 
B-7  of  Appendix  B  for  a  complete  listing).  Students  believed  that  a  variety  of 
lessons  was  provided  (3.49),  the  trainer  was  easy  to  use  (3.50)  and  comfortable 
(3.36),  and  that  training  time  was  mostly  spent  in  troubleshooting  (3.53).  There 
was  a  significant  difference  among  respondents  to  this  latter  statement,  however, 
as  Low  EEMT  (3.27)  students  were  less  likely  to  agree  than  High  EEMT  (3.79) 
students  [ t(91)  =  2.22,  f^.05].  Students  disagreed  with  statements  that  the 
lessons  were  too  long  (2.43)  and  that  training  time  on  the  equipment  was  often 
wasted  (2.47). 

Students  were  asked  to  provide  written  responses  concerning  their  likes, 
dislikes,  and  other  general  comments  about  the  2D  EEMT  training.  Because  of  the 
number  and  variety  of  these  responses,  a  complete  list  of  the  students'  likes  and 
dislikes  has  been  provided  in  Table  B-7  of  Appendix  B.  Two  general  observations 
are  possible  from  this  table:  (1)  the  dislikes  outnumber  the  likes  and  (2)  many 


EEMT  system  features  are  included  in  both  lists.  The  most  common  student  likes 
were  ease  of  use,  comfort,  reduced  safety  risk,  availability  for  practice 
troubleshooting  outside  of  scheduled  laboratory  periods,  and  ability  to  solve  more 
problems  in  a  given  period  of  time.  The  most  common  student  dislikes  were 
equipment  malfunctions  (particularly  touch  panel  drift  and  EEMT  lockup),  use  of 
test  equipment,  inability  to  access  components  properly  or  read  test  point 
designations,  and  the  possibility  of  developing  bad  safety  habits. 

The  general  comments  provided  by  students  were  far  more  consistent. 
Nineteen  students  remarked  that  the  EEMT  is  a  good  training  aid  but  it 
cannot/ should  not  replace  the  operational  equipment;  13  students  thought  the 
EEMT  should  be  used  for  remediation  and  extra  study;  six  students  preferred  more 
operational  equipment  training. 

Operational  equipment  vs.  2D  EEMT.  Four  rating  items  were  identical 


for  both  the  operational  equipment  and  the  2D  EEMT  training  sections  of  the 
student  critiques.  For  two  of  these  items,  students  rated  operational  training 
significantly  better  than  2D  EEMT  training.  Students  thought  that  training  time 
was  wasted  more  on  the  EEMT  (2.48)  than  on  the  operational  (1.67)  equipment 
[t(95)  =  5.40,  £<.001  ]  and  that  equipment  breakdowns  interfered  more  with 
training  on  the  EEMT  (1.60)  than  on  the  operational  (2.49)  equipment, 
(t(94)  =  5.91,  £<.001  ] .  With  respect  to  the  variety  of  lessons  provided,  however, 
the  EEMT  (3.94)  was  rated  slightly  higher  than  the  operational  (3.49)  equipment 
training  [t(88)  =  3.18,  £<.01]  .  This  finding  was  in  spite  of  the  fact  that  students 
commented  on  lesson  variety  as  a  feature  of  the  EEMT  training  that  they  disliked 
(rf.  Table  B-7  of  Appendix  B). 

Students  believed  that  the  2D  EEMT  can  be  a  useful  and  convenient 
training  aid  when  used  in  conjunction  with  the  operational  equipment.  However, 
because  access  to  the  operational  equipment  is  limited,  students  prefer  to 
maximize  their  operational  training  time  during  laboratory  periods  and  reserve 
EEMT  training  for  extra  study/remediation  settings.  The  factors  which  appear 
most  important  for  further  student  acceptance  are  (1)  increased  system 
reliability,  (2)  better  incorporation  of  safety  principles,  and  (3)  improved 
representations  of  test  points  and  test  equipment  use. 


Summary 


Overall,  instructors  and  students  were  quite  consistent  in  their  opinions 
of  the  2D  EEMT  system.  The  analysis  of  user  acceptance  determined  that: 

•  Both  groups  identified  an  important,  useful,  but  limited 
role  for  the  trainer  as  a  supplement  to  conventional 
AN/SPS-10  instruction. 

•  Extra  study/remediation  was  considered  to  be  the  most 
appropriate  setting  for  the  2D  EEMT  because  it  provided 
more  realistic  troubleshooting  experience  than  workbooks 
and  required  very  little  instructor  supervision. 

•  Both  groups  were  particularly  negative  about  the  low 
operational  reliability  of  the  EEMT  (e.g.,  touch  panel 
problems  and  system  lockup). 


V.  COST  ANALYSIS 


The  objectives  of  this  component  of  the  TCT  were  to  document  the 
investment,  operating,  and  maintenance  costs  of  the  2D  EEMT  system  and  to 
compare  the  life  cycle  costs  of  the  2D  EEMT  and  operational  equipment  when 
used  as  teaching  aids  in  the  Primary  Power  area  of  radar  systems  training  at  the 
ET  "A"  School.  The  major  hypothesis  considered  in  this  analysis  was: 

•  The  2D  EEMT  system  provides  electronics  training  at  less 
than  half  the  cost  needed  to  acquire,  operate,  and  maintain 
conventional  operational  equipment  to  accomplish 
equivalent  training  (Pine,  Daniels,  &  Malec,  1981). 

Additionally,  the  factors  which  affect  the  cost  of  using  2D  EEMT  systems  are 
identified  and  discussed.  To  illustrate  the  relationship  between  the  cost  of  using 
these  trainers  and  the  training  environment,  the  2D  EEMT  and  the  operational 
equipment  were  also  compared  in  selected  training  scenarios. 


Method 


In  order  to  assure  a  systematic  cost  analysis,  a  basic  framework  was 
developed  to  document  and  compare  the  life  cycle  costs  of  the  2D  EEMT  with 
those  of  operational  equipment.  This  framework  was  also  used  to  guide  the 
analysis  of  various  training  scenarios  in  which  the  2D  EEMT  was  considered  as  an 
alternative  or  supplemental  training  aid  to  current  practices. 

The  cost  framework  includes  five  categories:  facilities,  equipment, 
instructional  materials,  instructor  and  support  personnel,  and  students;  the 
subcategories  associated  with  each  major  category  are  as  follows: 

Facilities 

•  space  to  house  training  devices 

•  space/utilities  modifications  required 

Equipment 

•  hardware 

•  original  software  development 

•  system  specification  and  acquisition  management 

•  support  and  installation  of  trainers 

•  sustaining  investment  for  maintenance 

•  support  personnel 

Instructional  material 


•  lessonware  development  and  revisions 

•  training  manuals 


Personnel 


•  instructors'  wages 

•  overhead  burden 

Students 


•  wages 

•  billeting  and  other  support 

In  order  to  document  the  life  cycle  costs  of  the  2D  EEMT,  all  of  the 
major  cost  categories  were  used.  Further,  since  the  cost  of  system  design  is  a 
nonrecurring  cost  associated  only  with  the  prototype  device,  this  was  considered  a 
sunk  cost  in  the  life  cycle  cost  analysis.  System  design  costs  are,  however, 
presented  in  Table  9  where  prototype,  next  copy,  and  an  additional  copy  of  the  2D 
EEMT  are  compared. 

For  comparative  analyses  of  2D  EEMT  and  operational  equipment,  the 
assumption  of  equivalent  training  effectiveness  was  made  and  operating  costs 
were  estimated  over  a  15-year  life  cycle  period.  Ail  costs  were  adjusted  using 
the  annual  inflation  rate  to  reflect  1981  wholesale  commodity  prices.  To 
establish  net  present  value  (NPV)  in  1981  dollars,  operating  costs  were  discounted 
at  10  percent  annually  in  accordance  with  standard  Office  of  Management  and 
Budget  guidelines. 


Results 


2D  EEMT  Development  and  Acquisition  Costs 

Separate  acquisition  cost  estimates  were  made  for  the  prototype,  first 
copy,  and  an  additional  copy  of  the  2D  EEMT.  The  cost  of  the  prototype  system 
includes  development,  design  and  fabrication.  The  cost  of  the  first  copy  of  the 
trainer  includes  hardware,  operating  system,  and  documentation  but  excludes 
equipment  design  and  development.  The  cost  of  an  additional  copy  is  similar  to 
that  for  the  first  trainer  purchased,  except  that  some  cost  savings  are  realized 
assuming  the  device  is  used  for  the  same  training  application. 

The  summary  of  training  device  cost  estimates  presented  in  Table  9  was 
derived  from  contract  documents  supplied  by  the  2D  EEMT  contractor  and  by  the 
Navy  Personnel  Research  and  Development  Center  (NPRDC).  Appendix  C 
summarizes  contract  documents  used  in  this  analysis.  In  some  instances,  the  only 
cost  data  available  were  for  aggregated  expenditures  (e.g.,  system  and  user 
documentation)  or  included  expenditures  not  considered  relevant  to  the  TCT  (e.g., 
3D  EEMT  hardware  and  lessonware).  Generally,  supporting  documentation  was 
used  to  proportionally  allocate  aggregated  expenditures  to  the  appropriate  cost 
categories.  Irrelevant  costs  were  simply  noted  and  eliminated  from  this  analysis. 

Hardware  costs.  Table  9  shows  that  the  cost  of  the  various  hardware 
components  is  the  same  for  the  2D  EEMT  copies  and  the  prototype.  It  is  possible, 


Table  9 


2D  EEMT  Development  and  Acquisition  Costs 
(1981  Constant  Dollars) 

Training  Device  Cost  Notes 

Prototype  First  Copy  Additional  Copy  (rf.  Appen 
Cost  Categories _ 2D  EEMT  2D  EEMT _ •  2D  EEMT _ dix  D) 

Hardware 


Computer  (Altos) 

$  3,620 

$ 

3,620 

$ 

3,620 

Laser  Disk  Player 

2,500 

2,500 

2,500 

1 2"  Monitor  (Motorola) 

251 

251 

251 

1 2"  Touch  Panel  with  Bezel 

399 

399 

399 

19"  Monitor  (RCA) 

650 

650 

650 

19"  Touch  Panel  with  Bezel 

425 

425 

425 

Printer  (TI  Silent  700) 

974 

974 

0 

Station/Console 

1,151 

1,151 

1,151 

(A) 

System  Interfaces/Controllers 

1,314 

1,314 

1,314 

(B) 

Miscellaneous  Materials 

250 

250 

250 

(C) 

Fabrication 

6,777 

6,777 

6,672 

(D) 

Development 

195,237 

0 

0 

(E) 

User  Documentation 

34,549 

— 

1,727 

— 

0 

(F) 

Hardware  Total 

$  248,097 

$ 

20,038 

$ 

17,232 

(G) 

Software 

Development 

313,836 

0 

0 

(H) 

Acquisition 

0 

— 

50 

— 

50 

(I) 

Software  Total 

$  313,836 

$ 

50 

$ 

50 

Lessonware 

Development 

418,865 

125,000 

0 

(J) 

Acquisition 

0 

— 

637 

— 

637 

(K) 

Lessonware  Total 

$  418,865 

$  125,637 

$ 

637 

Logistics  Support 

Spares  package 

$  5,162 

$ 

5,162 

$ 

5,162 

(L) 

Total  System  Cost 


$  985,960  $  150,887  $  23,081 


however,  that  some  cost  savings  might  be  realized  if  a  sufficient  number  of 
trainers  were  purchased  at  the  same  time  to  obtain  a  discount  on  the  per  unit 
costs  of  each  component.  While  the  total  hardware  cost  of  only  the  prototype 
device  includes  system  and  user  documentation  development,  the  cost  of  acquiring 
user  documentation  already  developed  is  shown  as  an  expenditure  associated  with 
the  first  copy  purchased  for  a  specific  training  application.  The  costs  of  an 
instructor  terminal  (i.e.,  printer),  its  associated  fabrication  and  user 
documentation  are  not  shown  for  additional  2D  EEMTs.  For  purposes  of 
estimating  the  system  hardware  acquisition  cost,  it  is  assumed  that  the  instructor 
terminal  purchased  with  a  first  copy  of  the  trainer  can  support  four  additional 
units  used  for  the  same  training  application. 

Software  and  lessonware  costs.  The  costs  of  software  and  lessonware 
were  the  most  difficult  to  identify  for  a  number  of  reasons:  (1)  the  term  software 
was  frequently  used  in  contract  or  documentation  to  refer  to  both  operating 
system  software  and  lessonware;  (2)  lessonware  was  originally  developed  or 
proposed  to  be  developed  for  several  ET  "A”  School  areas,  including  generic  and 
representative  equipment;  (3)  some  lessonware  was  also  developed  for  the  EW 
School  at  Corry  Station,  Florida;  and  (4)  software  and  lessonware  were  created  to 
support  the  3D  EEMT  systems.  Only  software  and  lessonware  costs  incurred  to 
support  training  for  the  Primary  Power  area  of  radar  training  in  the  ET  "A"  School 
at  Great  Lakes  Training  Center  were  considered  relevant  for  the  TCT. 

The  total  cost  of  prototype  software  shown  in  Table  8  includes 
expenditures  for  government  furnished  software,  initial  system  software,  and 
software  changes  made  under  contract  modifications.  Total  software  costs  for  a 
first  copy  and  an  additional  2D  EEMT  do  not  include  any  development  effort  and 
are  shown  to  be  limited  to  the  purchase  of  diskettes.  Similarly,  the  total  cost  of 
prototype  lessonware  is  shown  to  include  government  furnished  lessonware,  initial 
2D  EEMT  lessonware,  modifications  to  lessonware  strategies  and  videodisc 
production.  Lessonware  cost  estimates  shown  in  Table  9  for  the  first  trainer  have 
been  made  assuming  that  a  typical  training  application  will  require  the  creation  of 
a  normal  operations  free-play  data  base  diskette  and  about  30  practice  problems 
and  the  production  and  mastering  of  a  supporting  videodisc.  Other  expenditures 
associated  with  the  first  copy  lessonware  include  student  diskettes,  and  printer 
paper.  Assuming  that  additional  trainers  are  purchased  with  the  first  unit  and 
used  for  the  same  training  application,  lessonware  expenditures  are  limited  to  the 
cost  of  videodisc  and  diskette  copies  and  printer  paper.  Note,  however,  that 
videodiscs  not  purchased  in  conjunction  with  the  mastering  process  will  require 
remastering  at  the  cost  of  $1,700. 

Logistics  support  costs.  The  cost  of  logistics  support  shown  in  Table  9 
was  derived  from  contract  documents  and  includes  the  spare  parts  package 
purchased  by  NPRDC  from  the  2D  EEMT  manufacturer  and  directly  from 
hardware  vendors.  These  expenditures  were  allocated  across  the  20  2D  EEMT 
units  acquired  under  the  Basic  Contract  to  obtain  the  per  unit  cost.  As  the 
number  of  devices  supported  increases,  it  might  be  expected  that  some  per  unit 
cost  savings  will  be  realized. 

Total  system  costs.  The  results  of  the  cost  analysis  shown  in  Table  9 
indicate  that  the  design  and  development  of  the  prototype  2D  EEMT  was  nearly  $1 


.v 


million  including  government  furnished  equipment,  software  and  lessonware. 
Total  system  costs  for  the  first  copy  and  an  additional  copy  were  $150,887  and 
$23,081,  respectively.  The  difference  of  $127,086  between  the  first  and 
additional  copies  of  the  trainer  is  accounted  for  by  two  major  factors:  (1)  the 
instructor  terminal  (e.g.,  printer)  and  related  system  support  and  (2)  lessonware 
development.  For  purposes  of  estimating  future  2D  EEMT  acquisition  costs,  it 
should  be  assumed  that  the  cost  of  lessonware  development  will  be  incurred  only 
once  for  each  training  application  and  that  one  instructor  terminal  (and  related 
costs  shown  under  hardware)  will  support  a  total  of  five  2D  EEMT  units. 


Life  Cycle  Cost  Analysis 

Tables  10,  11,  and  12  show  the  life  cycle  costs  associated  with  the  2D 
EEMT  first  copy,  an  additional  2D  EEMT,  and  the  AN/SPS-10  operational 
equipment,  respectively.  All  initial  investment  and  operating  costs  are  reported 
in  constant  1981  dollars;  operating  costs  are  projected  over  a  15-year  period  and 
include  expenditures  for  maintenance,  instructional  materials  and  instructor,  and 
student  support. 

Differences  in  facilities  cost  between  the  2D  EEMT  systems  (Tables  10 
and  11)  and  the  AN/SPS-10  (Table  12)  reflect  the  larger  amount  of  space  required 
to  support  each  AN/SPS-10  unit  as  compared  to  a  2D  EEMT  unit.  In  general, 
however,  no  alterations  to  existing  facilities  are  required  to  introduce  the  2D 
EEMT  system  into  a  typical  laboratory  training  environment. 


With  respect  to  equipment,  the  cost  of  2D  EEMT  hardware  is 
substantially  less  than  that  for  the  AN/SPS-10  operational  equipment.  Savings  of 
about  70  percent  over  the  cost  of  a  replacment  AN/SPS-10  unit  are  shown  with 
purchase  of  either  a  first  copy  or  additional  copy  of  the  2D  EEMT.  The  higher 
costs  for  sustaining  investment,  maintenance  and  operation  (power  requirements) 
of  the  AN/SPS-10  also  result  in  savings  with  the  purchase  of  a  2D  EEMT  system. 
This  is  expected,  however,  since  sustaining  investment  costs  are  calculated  as  a 
percentage  of  the  initial  hardware  cost.  Maintenance  costs  reflect  actual 
personnel  wages.  These  costs  differ  because  the  AN/SPS-10  equipment  is 
maintained  solely  under  a  service  contract,  whereas  the  2D  EEMT  is  maintained 
by  ET  "A”  School  personnel.  Lastly,  because  operating  costs  are  allocated  on  a 
per  square  foot  basis,  the  2D  EEMT  is  somewhat  less  expensive  to  operate  than 
the  actual  equipment. 

The  most  significant  cost  difference  between  the  2D  EEMT  and  the 
operational  equipment  is  the  lessonware  development  investment  cost  associated 
with  the  2D  EEMT  first  copy  (Table  10).  This  expenditure  reflects  the  cost  of 
developing  a  free-play  data  base  and  30  free-play  problems  used  with  the  2D 
EEMT  unit  in  the  ET  "A”  School  environment.  As  Table  11  shows,  however,  this 
cost  is  not  incurred  with  additional  2D  EEMTs  used  for  the  same  training 
application.  The  AN/SPS-10  operational  equipment  life  cycle  cost  (Table  12)  does 
not  include  an  investment  for  lessonware.  The  costs  of  developing  the  training 
problems  used  in  the  Primary  Power  radar  laboratory  were  considered  as  sunk 
costs  and  therefore,  not  estimated.  In  the  exisiting  environment,  only  updating  of 
training  problems  over  the  15-year  life  cycle  was  considered  necessary.  The 


constant  dollars 


Total  cost  in  constant  dollars  =  $1 1,612,680 
Net  present  value  (1981)  =  $5,923,248 


constant  dollars 


reoccurring  cost  of  printing  student  guides  (FOMMs)  was  estimated  to  be  the  same 
for  all  training  equipment.  Instructor  and  student  costs  are  the  same  for  the  2D 
EEMT  and  AN/SPS-10  systems  because  the  two  devices  were  used  in  a  similar 
manner  in  the  same  training  environment. 


A  comparison  of  the  total  life  cycle  costs  shown  in  Tables  10  and  12 
indicates  that  the  first  copy  of  the  the  2D  EEMT  system  will  cost  $84,608  more 
than  the  AN/SPS-10  operational  equipment  over  the  15-year  period  in  1981  NPV 
dollars.  Nearly  all  of  this  cost  difference  can  be  attributed  to  the  lessonware 
development  investment  cost  for  the  first  copy  of  the  2D  EEMT.  Comparison  of 
the  total  life  cycle  cost  of  an  additional  copy  of  the  2D  EEMT  (Table  11)  and  the 
AN/SPS-10  operational  equipment  reveals  a  savings  of  $43,182  with  the  2D  EEMT. 
Neither  of  these  comparisons,  however,  is  a  true  reflection  of  costs.  While  the 
lessonware  development  cost  associated  with  the  first  copy  of  the  trainer  is 
unavoidable,  it  is  a  nonrecurring  cost  if  multiple  2D  EEMTs  are  used  for  the  same 
training  application.  In  fact,  the  acquisition  of  three  2D  EEMTs  (a  first  copy  and 
two  additional  copies)  results  in  a  comparable  investment,  in  1981  NPV,  to  the 
acquisition  of  three  AN/SPS-10  units.  Each  additional  2D  EEMT  copy  acquired  at 
less  cost  than  an  additional  AN/SPS-10  unit,  results  in  an  overall  life  cycle  cost 
savings  of  $43,182. 


To  illustrate  this  point,  consider  the  ET  "A"  School  environment  where 
20  2D  EEMT  units  were  purchased  under  the  basic  contract.  Including  the  cost  of 
the  first  copy  of  the  EEMT,  three  trainers  with  instructor  terminals  and  required 
support  (but  excluding  lessonware  development)  and  16  additional  units,  the  life 
cycle  savings  over  purchase  of  20  AN/SPS-10  units  is  $1,150,804  in  1981  NPV. 


These  analyses  which  focus  primarily  on  single  units  show  that  the  major 
factor  contributing  to  the  potential  lower  life  cycle  cost  of  the  2D  EEMT,  as 
compared  to  operational  equipment,  is  the  initial  equipment  investment.  Thus,  in 
training  environments  where  a  single  operational  equipment  trainer  will  be 
acquired,  the  initial  investment  costs  must  exceed  about  $143,000^  or  require 
support  or  operating  expenditures  well  in  excess  of  those  for  the  AN/SPS-10  if  the 
2D  EEMT  is  to  show  a  lower  life  cycle  cost  estimate.  Alternately,  if  less 
expensive  operational  equipment  is  necessary  to  support  training,  the  2D  EEMT 
would  show  a  lower  life  cycle  cost  estimate  only  if  multiple  units  were  required  or 
the  operational  equipment  had  associated  support  or  operating  costs  substantially 
higher  than  those  for  the  AN/SPS-10. 


In  sum,  examination  of  the  life  cycle  cost  estimates  shown  in  Tables  10, 
11,  and  12  indicates  that  acquisition  of  a  2D  EEMT  will  result  in  a  slightly  higher 
cost  for  a  first  copy  and  a  slightly  lower  cost  for  an  additional  copy  than  the 
acquisition  of  an  AN/SPS-10  operational  unit.  When  multiple  units  are  needed  and 
investment  costs  are  spread  over  additional  copies,  acquisition  of  the  2D  EEMT 
results  in  a  life  cycle  cost  savings  over  the  AN/SPS-10  when  three  copies  are 
acquired. 


^This  estimate  was  derived  by  adding  the  acquisition  cost  of  the 
AN/SPS-10  and  difference  in  the  life  cycle  costs,  in  1981  NPV,  of  this  device  and 
the  first  copy  of  a  2D  EEMT  (rf.  Tables  10  and  12). 


A 


A  A  A 


Considering  only  initial  investment  costs,  the  purchase  of  20  2D  EEMT 
units  results  in  a  cost  savings  of  approximately  50  percent  over  the  purchase  of  20 
AN/SPS-1 0  units.  This  comparison  is  based  on  the  assumption  that  four  instructor 
terminals  will  be  needed  to  support  the  20  2D  EEMT  units  and  the  one  time  cost 
of  trainer  lessonware  will  be  incurred.  Since  it  was  not  possible  to  estimate  the 
cost  lessonware  acquisition  for  the  AN/SPS-10  units,  the  estimate  of  20  units  to 
attain  a  50  percent  cost  savings  is  conservative. 


Hypothetical  Training  Scenarios 


Since  the  actual  life  cycle  cost  savings  that  might  be  realized  by 
employing  2D  EEMT  units  will  depend  in  large  part  on  the  training  needs  and 
environments  in  question,  the  following  scenarios  are  presented  to  illustrate  how 
the  preceding  cost  data  might  be  used  to  assist  in  the  equipment  acquisition 
decision  making  process.  Further,  because  many  training  environments  are 
similar  to  that  of  the  ET  "A"  School  at  Great  Lakes  Naval  T raining  Station,  four 
of  the  examples  focus  on  situations  involving  the  addition  or  modification  of 
multiple  student  training  stations  in  this  typical  environment.  The  fifth  scenario 
involves  transferring  this  training  approach  to  another  environment. 


Scenario  1:  Comparison  of  Supplemental  Trainer  Options 

In  this  scenario,  it  is  assumed  that  it  is  necessary  to  increase  the  number 
of  training  devices  to  meet  an  increased  demand  on  the  ET  "A"  School.  Table  13 
shows  the  comparison  of  training  costs  associated  with  acquiring  10  additional 
operational  equipment  units  (AN/SPS-10  or  AN/SPS-67)  or  2D  EEMT  units  to 
expand  the  capability  of  the  Primary  Power  section  of  the  Radar  Systems  course. 
The  AN/SPS-67  was  selected  as  a  comparative  operational  system  because  this 
radar  unit  is  the  most  likely  replacement  for  the  AN/SPS-10  in  the  fleet  and 
would  probably  replace  the  AN/SPS-10  as  the  operational  equipment  trainer  in  the 
ET  "A"  School  as  well. 

The  characteristics  of  the  ET  "A"  School  environment  were  used  to 
establish  the  parameters  of  the  comparison.  First,  it  was  assumed  that  the 
existing  laboratory  housed  10  actual  equipment  trainers  and  an  additional  10 
stations  (operational  equipment  or  2D  EEMTs)  were  needed  to  manage  an 
increased  student  flow.  Additional  trainers  for  extra  study/remediation  purposes 
were  not  included  in  this  analysis.  Second,  student  flow  was  assumed  to  be  66  per 
class,  or  approximately  22  students  per  shift  per  week.  Third,  for  safety  reasons, 
it  was  necessary  to  have  one  instructor  present  in  the  laboratory  for  every  10 
training  stations  in  use. 

As  shown  in  Table  13,  the  life  cycle  costs  associated  with  adding  10  2D 
EEMT  units  in  the  laboratory  training  environment  are  less  than  those  associated 
with  adding  10  AN/SPS-10  units  or  introducing  10  newer  AN/SPS-67  units.  The 
NPV  and  cost  of  training  per  student-hour  calculations  also  indicate  a  cost  savings 
for  the  2D  EEMT  over  both  the  AN/SPS-10  and  AN/SPS-67  options.  To  illustrate 
the  extent  of  this  potential  cost  savings,  consider  a  training  environment  involving 


Table  13 


Scenario  1 :  Life  Cycle  Cost  Comparison  of 
Supplemental  Trainer  Options 


Cost 

Categories 

2D  EEMT 
(10  units) 

Training  Device 

AN/SPS-10 
(10  units) 

AN/SPS-67 
(10  units) 

Cost  Notes 
(rf.  Appendix  t 

Facilities 

$  90,000 

$  180,000 

$  180,000 

(M) 

Equipment 

Hardware 

$  177,932 

$  581,300 

$1,400,000 

(AA),  (BB) 

Software 

500 

0 

0 

(0 

Sustaining  Investment 

201,620 

435,000 

1,050,000 

(O) 

Maintenance 

136,500 

324,000 

324,000 

(P),  (Y) 

Operating  Costs 

27,000 

42,000 

42,000 

(Q),  (Z) 

Instructional  Materials 

TOs/Text  Lessons 

$1,063,578 

$  949,500 

$  983,089 

(3),  (R),  (CC) 

Videodisc  Development 

10,922 

0 

0 

(3) 

Copies 

170 

0 

0 

(K) 

Diskettes/Printer  Paper 

6,200 

0 

0 

(K) 

Updating 

5,300 

5,300 

5,300 

Personnel 

$  425,200 

$  425,200 

$  425,200 

(T>,  (U) 

Students 

$10,167,000 

$10,167,000 

$10,167,000 

(V),  (W) 

Total  Cost 

$12,311,922 

$13,109,300 

$14,576,589 

NPV  1981 

$6,426,108 

$6,939,010 

$8,103,317 

Cost  per  student-hour 

$  60.85 

$  65.71 

$  76.74 

60  students  per  week.  If  10  2D  EEMT  units  were  used  instead  of  10  AN/SPS-10 
units,  a  cost  saving  of  $11,664  per  40  hour  week  would  be  realized.  The 
implication  of  this  alternative  cost  scenario  is  that  2D  EEMT  systems,  when 
purchased  in  multiple  copies  to  meet  expanded  training  requirements,  can  result 
in  substantial  cost  savings  over  the  purchase  of  additional  operational  equipments. 


Scenario  2:  Extensive  Modification  to  an  Existing  Trainer 

In  this  scenario,  it  is  assumed  that  10  2D  EEMT  units  have  been 
purchased  and  implemented  in  the  Primary  Power  radar  training  environment.  In 
addition,  it  is  assumed  that  the  existing  operational  equipment  used  as  trainers 
(e.g.,  the  10  AN/SPS-10  units  in  the  Primary  Power  laboratory)  require  extensive 
modifications  due  to  technology  changes.  Depending  on  the  extent  of 
modifications  to  the  operational  equipment,  modifications  to  the  2D  EEMT 
training  system  may  also  result.  Table  14  shows  the  estimated  cost  of 
modifications  to  the  2D  EEMT  if  changes  in  the  operational  equipment 
necessitated  development  of  15  new  free-play  lessons,  redevelopment  of  technical 
documentation  (FOMMs)  and  redevelopment  of  the  videodisc  image  base.  This 
estimate  represents  the  highest  2D  EEMT  modification  cost  possible  since 
complete  redevelopment  of  the  videodisc  image  base  would  not  be  necessary  in  all 
instances.  Equipment,  facilities,  personnel  and  student  costs  associated  with  the 
trainer  would  not  change. 

The  costs  associated  with  changes  in  the  operational  equipment  include 
equipment  modification  costs  estimated  at  one-half  the  cost  of  new  units,  plus 
twice  the  annual  cost  of  printing  technical  documentation  and  instructional 
materials  modification  costs  estimated  at  one-half  the  costs  of  developing 
lessonware.  Table  14  shows  that  when  operational  equipment  modifications  are 
necessary,  the  associated  costs  could  be  more  than  50  percent  if  2D  EEMT  units 
were  being  used  in  the  training  environment. 

The  need  for  very  extensive  modifications  to  the  operational  equipment 
could  result  in  replacement  of  existing  operational  equipment  units  with  a  new 
operational  system,  such  as  the  AN/SPS-67.  The  acquisition  costs  of  10  of  these 
units  was  shown  in  Table  13.  The  only  additional  impact  of  such  a  replacement  to 
the  2D  EEMT  uiits  would  be  full  lessonware  development  (30  free-play  lessons)  vs. 
1 5  free-play  lessons  as  described  earlier.  In  this  situation,  modification  of  the  2D 
EEMT  imits  would  result  in  dramatic  savings  over  the  cost  of  acquiring  new 
operational  equipment. 

This  scenario  illustrates  that  the  decision  to  acquire  10  2D  EEMT  units 
can  result  in  additional  savings  if  replacement  or  modifications  to  the  existing 
operational  units  used  for  training  should  be  required  at  a  later  date. 


Scenario  3:  Classroom  Use 

Introduction  of  the  2D  EEMT  system  into  the  classroom,  e.g.,  the 
AN/SPS-10  Primary  Power  classroom,  would  result  in  expenditures  for  additional 
trainers  and  structured  lessonware  development.  These  costs  are  shown  in  Table 
1 5.  Facilities  modifications  would  be  minimal  and  therefore  are  not  considered  an 


Table  1 4 


Scenario  2:  Cost  Implications  for  the  2D  EEMT  and 
Operational  Equipment  due  to  Extensive 
Modifications  to  Operational  Equipment 


Cost 

Categories 

Equipment  Cost 

2D  EEMT  AN/SPS-10 

Cost  Notes 
(rf.  Appendix  D) 

Facilities 

NC* 

NC 

Equipment 

Hardware 

NC 

$290,650 

Software 

NC 

- 

Sustaining  Investment 

NC 

NC 

Maintenance 

NC 

NC 

Operating  Costs 

NC 

NC 

Instructional  Materials 

TOs/Text  Lessons 

$183,539 

$147,595 

(DD) 

Videodisc  Development 

10,922 

- 

Copies 

170 

- 

Diskettes/Printer  Paper 

NC 

- 

Updating 

NC 

NC 

Personnel 

NC 

NC 

Students 

NC 

NC 

Total  Cost  of  Modifications 

(1981  dollars) 

$194,631 

$438,245 

*No  change  from  Table  13 


Table  15 


Scenario  3:  Cost  Implications  of  Introducing 
2D  EEMT  into  the  Classroom 


Cost 

Categories 

Cost 

Implications 

Cost  Notes 
(rf.  Appendix  D) 

Facilities 

NC 

Equipment 

Hardware 

$88,966 

(EE) 

Software 

250 

Sustaining  Investment 

30,810 

Maintenance 

NC 

Operating  Costs 

NC 

Instructional  Materials 

TOs/Text  Lessons 

$96,525 

(FF) 

Videodisc  Development 

1,700 

(3) 

Copies 

85 

(K) 

Diskettes/Printer  Paper 

620 

(K) 

Updating 

NC 

Personnel 

NC 

Students 

NC 

Total  Cost  of  Introduction 
into  Classroom 


$218,956 


additional  cost.  Five  2D  EEMT  units  would  be  sufficient  to  meet  classroom 
training  objectives  since  students  could  be  rotated  between  workbook  activities 
and  2D  EEMT  lessons.  Additional  maintenance  and  operating  costs  are  considered 
minimal. 

Structured  lessonware  would  be  more  appropriate  to  classroom  use  than 
free-play  lessonware,  since  it  focuses  on  overall  AN/SPS-10  operational 
equipment  familiarity  rather  than  the  troubleshooting  practice.  No  alterations  to 
the  videodisc  image  base  would  be  necessary  for  use  in  the  Primary  Power 
classroom.  Additional  copies  of  the  videodisc,  however,  would  require 
remastering  of  the  disc.  No  additional  units  in  extra  study/remediation  are 
projected;  the  shared  use  of  extra  study/remediation  units  for  classroom  and 
laboratory  extra  study/remediation  is  assumed.  However,  structured  lessonware 
for  use  in  the  extra  study/remediation  laboratory  would  be  necessary.  Costs  for 
duplication  of  this  lessonware  would  be  minimal.  Table  15  shows  that  the  total 
equipment  and  instructional  materials  costs  of  adding  five  2D  EEMTs  to  a 
classroom  setting  is  about  $  218,956. 

Introducing  operational  equipment  into  the  classroom  was  viewed  as  an 
inappropriate  use  of  the  equipment  due  to  space  and  safety  considerations. 
Therefore,  this  scenario  shows  the  costs  involved  in  expanding  the  use  of  the  2D 
EEMT  into  a  classroom,  a  training  environment  for  which  operational  equipment  is 
not  particularly  appropriate. 


Scenario  Changes  in  Skill  Level  Training 

The  2D  EEMT  could  be  considered  for  training  at  skill  levels  other  than 
"A"  School,  including  basic  electronics  or  advanced  "C"  School  training,  since  the 
hardware  and  software  configurations  can  support  a  variety  of  training 
applications  and  environments.  Use  of  the  2D  EEMT  for  more  or  less  skilled 
training  areas  would  necessitate  acquisition  of  additional  units  appropriate  to  the 
extent  of  training  planned  and  estimated  student  flow.  Extensive  facilities 
alterations  would  not  be  required  to  accommodate  the  2D  EEMT  units.  Hardware 
and  software  modifications  would  not  be  required.  Development  of  lessonware  at 
an  appropriate  skill  level  would  be  required  and  a  new  videodisc  image  base  would 
be  needed  for  the  equipment  in  question.  The  number  of  videodisc  copies  required 
would  depend  on  the  number  of  2D  EEMT  units  installed.  Training  of  instructors 
to  operate  2D  EEMT  at  other  training  levels  would  be  necessary  and  student  costs 
would  vary  according  to  student  flow  and  to  the  length  of  the  training  program. 
The  acquisition  and  life  cycle  costs  associated  with  the  application  of  2D  EEMT 
units  in  either  of  these  environments  would  be  comparable  to  those  shown  in 
Tables  9  and  13,  respectively,  with  adjustments  for  the  actual  number  of  units  to 
be  acquired. 

The  costs  of  implementing  this  scenario  would  be  relatively  high  since 
complete  lessonware  and  videodisc  development  costs  would  be  incurred  along 
with  acquisition  equipment  costs.  The  justification  for  implementing  2D  EEMT 
units  in  these  environments  is  the  expansion  of  the  training  concept  to  new 
training  areas  and  the  ease  of  modification  for  future  operational  equipment 
modifications.  Because  of  the  many  variables  described  and  the  uncertainty  of 


the  alternative  training  equipment  options  available,  no  cost  implications  table 
has  been  developed  for  this  scenario. 


Scenario  5:  Change  in  Physical  Training  Environment 

This  scenario  is  similar  to  Scenario  4  except  that  a  physical  change  of 
environment  rather  than  skill  level  change  is  considered.  In  Scenario  5,  the  2D 
EEMT  is  implemented  in  a  nonclassroom  environment,  such  as  shipboard  or  at  a 
dock-side  facility.  The  introduction  of  the  2D  EEMT  into  these  environments 
presents  a  particular  challenge.  The  cost  of  changes  to  the  hardware  or  the 
system  configuration  required  to  meet  nonclassroom  or  nonlaboratory 
environments  cannot  be  estimated  from  the  present  cost  analysis.  Facilities 
modifications,  maintenance  and  operating  costs  would  also  vary  from  costs 
previously  presented.  Full  lessonware  development,  including  videodisc 
production,  would  be  required  unless  the  AN/SPS-10  operational  equipment  was 
the  target  of  the  training  effort.  Personnel  and  student  costs  would  vary  with 
student  flow  and  the  length  of  training.  This  scenario  is  included  only  to  suggest 
that  the  instructional  approach  used  by  the  2D  EEMT,  with  appropriate;  hardware 
modifications,  might  be  adaptable  to  a  range  of  diverse  training  environments. 
This  possibility  is  supported  by  the  results  of  Table  13  which  indicate  that 
expanded  use  of  the  2D  EEMT,  as  opposed  to  purchase  of  additional  operational 
units,  is  more  cost  effective  when  training  effectiveness  is  at  least  equivalent. 


Summary 


The  analysis  of  the  investment,  operating,  and  maintenance  costs  of  the 
2D  EEMT  indicates  that: 

•  When  single  units  are  purchased,  acquisition  of  a  2D  EEMT 
will  result  in  a  slightly  higher  cost  for  a  first  copy  and  a 
slightly  lower  cost  for  an  additional  copy  than  the 
acquisition  of  an  AN/SPS-10  operational  unit.  Nearly  all 
of  the  cost  difference  can  be  attributed  to  the  lessonware 
development  investment  cost  for  the  first  copy  of  the  2D 
EEMT.  Because  a  similar  estimate  of  lessonware  costs  for 
the  AN/SPS-10  was  not  available,  the  apparent  cost 
difference  is  somewhat  inflated. 

•  When  multiple  units  are  purchased,  the  2D  EEMT  results  in 
a  life  cycle  cost  savings  over  the  AN/SPS-10  when  three 
copies  are  acquired.  However,  to  realize  a  life  cycle  cost 
savings  of  50  percent,  it  would  be  necessary  to  purchase 
over  100  2D  EEMT  units  instead  of  an  ec^ial  number  of 
AN/5PS-10  units. 

•  Considering  only  initial  investment  costs,  the  purchase  of 
20  2D  EEMT  units  results  in  a  cost  savings  of 
approximately  50  percent  over  the  purchase  of  20 
AN/SPS-10  units.  Since  it  was  not  possible  to  estimate  the 


cost  lessonware  acquisition  for  the  AN/SPS-10  units,  the 
estimate  of  20  units  to  attain  a  50  percent  cost  savings  is 
conservative. 


•  The  actual  life  cycle  cost  savings  realized  by  using  2D 
EEMT  units  instead  of  operational  equipment  depends  in 
large  part  on  the  initial  investment  cost  of  the  operational 
equipment.  In  training  environments  where  a  single 
operational  equipment  trainer  will  be  acquired,  the  initial 
investment  costs  must  exceed  about  $1  <>3,000  or  require 
support  or  operating  expenditures  well  in  excess  of  those 
for  the  AN/SPS-10  if  the  2D  EEMT  is  to  show  a  lower  life 
cycle  cost  estimate.  Alternately,  if  less  expensive 
operational  equipment  is  necessary  to  support  training,  the 
2D  EEMT  would  show  a  lower  life  cycle  cost  estimate  only 
if  multiple  units  were  required  or  the  operational 
equipment  had  associated  support  or  operating  costs 
substantially  higher  than  those  for  the  AN/SPS-10. 

•  Considering  the  training  costs  per  student-hour,  a 
substantial  savings  can  be  realized  by  use  of  2D  EEMT 
units  rather  than  operational  equipment. 

•  The  cost  of  modifying  training  equipment  to  accommodate 
technological  changes  in  operational  systems  can  be 
reduced  by  more  than  50  percent  if  2D  EEMT  units  are 
already  in  use  in  the  training  environment. 


VI.  GENERAL  CONCLUSIONS  AND  RECOMMENDATIONS 


The  findings  of  the  TCT  suggest  some  general  conclusions  regarding  the 
training  effectiveness,  operational  suitability,  and  costs  of  the  2D  EEMT.  This 
section  summarizes  these  conclusions  and  presents  recommendations  related  to 
each  of  these  three  areas  of  study. 


Training  Effectiveness 


Conclusions 


•  When  used  in  conjunction  with  actual  equipment  trainers, 
the  2D  EEMT  can  reduce  reliance  on  operational  equipment 
trainers  without  a  reduction  in  student  troubleshooting 
performance. 

•  The  use  of  the  2D  EEMT  is  equally  effective  for  low, 
middle,  and  high  ability  students. 

•  The  2D  EEMT  is  an  effective  alternative  to  the  use  of 
operational  equipment  for  providing  extra  study  and 
remediation  opportunities. 

•  When  introduced  into  a  training  laboratory  setting,  the  2D 
EEMT  provides  an  increased  opportunity  for 
troubleshooting  practice  on  an  individual  basis,  since  safety 
and  instructor  monitoring  requirements  are  substantially 
reduced. 

•  The  skills  and  knowledge  learned  on  the  EEMT  system 
generalize  to  the  larger  population  of  equipment 
maintained  by  ETs. 


Recommendations 

•  The  2D  EEMT  should  be  used  to  supplement  operational 
equipment  in  training  environments  which  serve  large 
student  populations.  Using  the  trainers  to  reduce  reliance 
on  operational  equipment  by  as  much  as  50  percent  would 
increase  the  student-equipment  contact  time  and  allow  a 
higher  student-instructor  ratio,  while  maintaining  a  safe 
training  environment. 


•  The  use  of  the  2D  EEMT  for  extra  study/ remediation 
training  should  be  encouraged.  With  the  development  of 
appropriate  lessonware,  trainers  allocated  for  this  purpose 
could  support  a  variety  of  courses  with  relatively  little 
instructor  supervision. 
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•  Alternative  techniques  for  integrating  the  2D  EEMT  into 
existing  training  environments  could  result  in  observable 
increments  in  student  performance  and  should  be 
examined.  EEMT  lessonware  should  support  hands-on 
training  by  providing  the  necessary  background  information 
and  the  opportunity  to  explore  alternative  solution 
strategies  in  a  safe  environment  rather  than  duplicate 
hands-on  experience  acquired  on  operational  equipment. 

•  Potential  applications  for  the  2D-3D  EEMT  system  should 
be  identified  and  investigated.  It  is  conceivable  that 
training  effectiveness  might  be  substantially  improved  by 
using  the  integrated  EEMT  system  since  it  would  fill  the 
gap  between  classroom  and  hands-on  operational  equipment 
training.  BEflcE  training  would  serve  as  an  excellent  test 
environment  since  relatively  naive  students  would  have  an 
opportunity  to  acquire  generic  hands-on  experience  in  a 
safe  environment. 


Suitability 


Conclusions 


•  While  the  2D  EEMT  is  generally  reliable,  40  percent  of  the 
118  required  maintenance  actions  were  related  to  touch 
panel  malf motions. 

•  The  2D  EEMT  was  available  for  training  about  72  percent 
of  the  time,  when  training  is  defined  as  laboratory  and 
extra  study/remediation. 

•  The  spares  package  for  the  2D  EEMT  available  during  the 
TCT  was  not  adequate  to  maintain  a  90  percent  laboratory 
training  availability. 

•  The  maintainability  of  the  2D  EEMT  system  was 
substantially  affected  by  touch  panel  malfunctions.  These 
panels  were  difficult  to  obtain  and  were  replaced  with  a 
newer  version  which  often  resulted  in  delays  of  3  to  6 
months. 

•  The  2D  EEMT  could  not  be  adequately  maintained  and 
logistically  supported  by  Navy  personnel  at  the  ET  "A" 
School,  given  the  limited  availability  of  spare  parts  during 
the  TCT.  Failures  in  major  system  components  (e.g., 
microprocessor,  videodisc  player)  are  best  resolved  by  the 
manufacturers. 

•  The  2D  EEMT  can  be  operated  more  safely  by  untrained 
students  than  operational  equipment. 
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•  The  2D  EEMT  is  easily  transported,  assembled,  and 
disassembled.  It  is  significantly  more  mobile  than 
operational  equipment. 

•  The  2D  EEMT  is  physically,  functionally,  and  electrically 
compatible  with  the  ET  School  environment. 

•  The  tutorial  lessonware  has  not  been  used.  It  was 
considered  too  lengthy  and  inflexible  in  presentation.  The 
potential  utility  of  the  2D  EEMT  in  a  classroom  setting  has 
not  been  demonstrated. 


The  2D  EEMT  lessonware  currently  in  use  does  not 
adequately  train  safety  procedures.  This  modification 
would  not  be  difficult  to  make,  however. 


•  Given  the  generic  nature  of  the  2D  EEMT,  it  is  readily 
adaptable  to  some  training  environments  with  only  changes 
in  lessonware  or  the  picture  data  base.  The  approach  to 
training  employed  by  the  2D  EEMT  may  be  appropriate  for 
an  even  wider  range  of  training  environments,  although 
hardware  modifications  would  also  be  required. 


•  The  data  management  features  of  the  2D  EEMT  were  not 
fully  examined.  Those  which  existed  during  the  TCT  were 
extremely  limited.  Improvements  in  the  software  are 
needed  to  increase  the  utility  of  student  performance 
records. 


•  Instructors  were  generally  supportive  of  including  the  2D 
EEMT  in  the  ET  "A"  School  environment,  although  it  was 
not  viewed  as  a  replacement  for  operational  equipment 
trainers. 


Students  were  noncommittal  with  respect  to  the  2D  EEMT. 
While  they  reported  that  the  trainer  provided  a  greater 
opportunity  to  work  on  troubleshooting  problems,  working 
on  the  operational  equipment  was  preferred. 


Recommendations 


System  malfunctions  resulting  f  rom  faulty  touch  panels  and 
inadequate  ventilation  for  equipment  components  in  the 
student  console  should  be  remedied  to  increase  the 
reliability  of  the  2D  EEMT. 


•  A  comprehensive  spare  parts  package  should  be  made 
available  on-site  to  increase  the  maintainability  of  the  2D 
EEMT  system.  The  package  should  be  developed  in  view  of 
the  maintenance  records  to  date. 
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•  The  2D  EEMT  lessonware  should  be  modified  to  monitor 
safe  operating  procedures  and  provide  timely  feedback  in 
the  form  of  printed  warnings,  graphic  displays,  or  audible 
alarms  in  response  to  student  safety  violations. 

•  The  2D  EEMT  software  should  be  modified  in  cooperation 
with  school  personnel  to  provide  student  performance  data 
that  is  more  timely,  comprehensive,  and  easy  to 
understand.  This  information  should  assist  instructors  so 
that  laboratory  and  extra  study/remediation  can  be 
tailored  to  meet  individual  and  class  training  needs. 

•  The  possibility  of  using  the  2D  EEMT  in  other  training 
environments  at  the  "A"  School  level  should  be  examined. 
In  a  classroom  setting,  lessonware  should  emphasize 
equipment  demonstration,  manipulation  and  familiariza¬ 
tion,  rather  than  troubleshooting. 


•  When  single  units  are  purchased,  acquisition  of  a  2D  EEMT 
will  result  in  a  slightly  higher  cost  for  a  first  copy  and  a 
slightly  lower  cost  for  an  additional  copy  than  the 
acquisition  of  an  AN/SPS-10  operational  unit.  Nearly  all 
the  cost  difference  can  be  attributed  to  the  lessonware 
development  investment  cost  for  the  first  copy  of  the  2D 
EEMT.  Because  a  similar  estimate  of  lessonware  costs  for 
the  AN/SPS-10  was  not  available,  the  apparent  cost 
difference  is  somewhat  inflated. 

•  When  multiple  units  are  purchased,  the  2D  EEMT  results  in 
a  life  cycle  cost  savings  over  the  AN/SPS-10  when  three 
copies  are  acquired.  However,  to  realize  a  life  cycle  cost 
savings  of  50  percent,  it  would  be  necessary  to  purchase 
over  100  2D  EEMT  units  instead  of  an  equal  number  of 
AN/SPS-10  units. 

•  Considering  only  initial  investment  costs,  the  purchase  of 
20  2D  EEMT  units  would  result  in  at  least  a  50  percent  cost 
savings  over  the  purchase  of  20  AN/SPS-10  units. 

•  The  actual  life  cycle  cost  savings  realized  by  using  2D 
EEMT  units  instead  of  operational  equipment  depends  in 
large  part  on  the  initial  investment  cost  of  the  operational 
equipment. 

•  Considering  the  training  costs  per  student-hour,  a 
substantial  savings  can  be  realized  by  use  of  2D  EEMT 
units  rather  than  operational  equipment. 
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•  The  cost  of  modifying  training  equipment  to  accommodate 
technological  changes  in  operational  systems  can  be 
reduced  by  more  than  50  percent  if  2D  EEMT  units  are 
already  in  use  in  the  training  environment. 


Recommendations 


•  Because  cost  savings  are  associated  with  multiple  2D 
EEMT  acquisitions,  these  trainers  are  recommended  for 
school  environments  with  relatively  high  enrollment  and 
diverse  training  applications. 

•  The  potential  for  use  of  the  2D  EEMT  system  in 
conjunction  with  equipment  that  is  more  costly  and 
sophisticated  than  the  AN/SPS-10  radar  should  be 
examined  since  additional  cost  savings  might  be  realized. 
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section  a-i 


FOLLOW-UP  RATING  FORM 


This  forrr,  is  to  be  used  to  rate  the  C  School 
performance  of  the  technician  named  in  the  box 
on  the  right.  The  information  contained  on  this 
form  will  in  no  way  be  used  to  affect  the 
technician's  personal  or  work  records,  or 
enhance  or  impede  his/her  career  status.  This 
information  is  being  collected  to  monitor  the 
C  School  performance  of  those  technicians 
whose  training  was  monitored  during  their  FT 
A  School  training  at  Great  Lakes. 


Lame  and  position  of  person  completing  this  f 

About  how  long  (in  months)  has  the  technician  been  at  this  school?  _ _ 

Compared  to  the  other  technicians  in  your  classes,  please  indicate  how  you  would 
rate  this  technician  in  each  of  the  following  areas: 

One  of  the  Below  Above  One  of  the 


attention  to  safety? 

Worst 

Avers ae 
2  ' 

Average 

3 

Averaae 

4 

Best 

5 

2 . 

ability  tu  troubleshoot? 

1 

2 

3 

4 

5 

2. 

use  of  equipment  manuals? 

i 

2 

3 

4 

t 

t 

-r  . 

ability  to  work  alone? 

i 

rs 

c 

3 

4 

5 

r 

performance  on  written  tests? 

2 

c 

3 

4 

5 

-  • 

performance  on  hands-on  tests? 

2 

2 

3 

4 

5 

- 

overall  C  School  performance? 

2 

2 

3 

4 

5 

Abe 

ut  how  many  hours  of  remediation 

or  extr-c 

study  have 

been  assigned  to  this 

"  A  ~ 

nnician  so  far?  is 

that  less 

than  usua i 

,  about  average,  or 

more 

than  usual? 


Technician  Name: 


C  School  Assignment: 


Identifying  Code: 


you  nave  any  additional  comments  about  the  performance  of  this  technician 
School? 


-2 


aeciioii  a -jl 


STUDENT  CRITIQUE 


Name : 


Class : 


It  is  requested  that  you  take  a  few  minutes  of  your  time  to  help  make  Radar 
training  more  effective.  This  information  is  for  evaluation  purposes  only 
and  will  be  kept  strictly  confidential. 

Circle  the  cumber  that  best  expresses  how  much  you  agree  with  each 
statement  below:  1  =  Not  At  All  and  5  =  Very  Much. 

*  -k'k'k'k'k'k'k'k'k'k'k’k'k'k 

I.  From  your  experience  with  the  operational  AN/SPS-10  radar  equipment,  to 
what  extent  would  you  agree  with  the  following  statements: 


Not 

At 

All 

Very 

Much 

Don' 

Know 

a . 

There  was  enough  training  time 
on  the  equipment 

1 

2 

3 

4  5 

b. 

Training  time  on  the  equipment  was 
often  wasted 

1 

2 

3 

4  5 

? 

c . 

A  variety  of  training  problems  were 
provided 

1 

2 

3 

4  5 

7 

d. 

Equipment  breakdowns  interfered  with 
training 

1 

2 

3 

4  5 

? 

e . 

Training  time  was  mostly  spent  on 
troubleshooting 

1 

2 

3 

4  5 

7 

f . 

More  time  should  be  spent  on  trouble¬ 
shooting 

1 

2 

3 

4  5 

7 

g- 

I  have  a  good  understanding  of  how  to 
maintain  this  equipment 

1 

2 

3 

4  5 

7 

h. 

I  feel  comfortable  working  on  this 

1 

2 

3 

4  5 

? 

equipment 
i . 


Any  additional  comments  about  your  training  that  involved  the  AN/SPS-10 
equipment? 


From  your  experience  with  the  2D  EEMT  subsystem  (equipment  pictures),  to 
what  extent  would  you  agree  with  the  following  statements  (if  you  have 
had  no  experience  please  skip  this  section): 

Not  Very 

At  All  Much 


a. 

The  lessons  were  useful  for 
laboratory  instruction 

1 

2 

3 

4 

5 

b. 

The  lessons  were  useful  for 
remedial  instruction 

1 

2 

3 

4 

5 

c. 

The  lessons  were  too  simple 

1 

2 

3 

4 

5 

d. 

The  lessons  were  too  long 

1 

2 

3 

4 

5 

e. 

A  variety  of  lessons  were  provided 

1 

2 

3 

4 

5 

f. 

The  equipment  was  easy  to  use 

1 

2 

3 

4 

5 

g- 

The  equipment  was  comfortable  to  use 

1 

2 

3 

4 

5 

h. 

Training  time  was  mostly  spent  on 
troubleshooting 

1 

2 

3 

4 

5 

i. 

Training  time  on  the  equipment  was  often 
wasted 

1 

2 

3 

4 

5 

Any  other  comments  about  your  training  on  the  2D  EEMT  subsystem? 


c 


v.^ 


v 


Section  A- 4 


INSTRUCTOR  CRITIQUE 


I.  About  how  long  (in  months)  have  you  been  an  instructor  at  Great  Lakes 
Naval  Training  Station?  _ 

About  how  long  (in  months)  have  you  been  an  instructor  in  the  following 
areas?  Please  mark  your  current  teaching  assignment  with  an  asterisk 
(*). 

Remediation  Classroom  Laboratory 

Communications  Systems:  _  _  _ 

Radar  Systems: 

Power  (4.1)  _  _  _ 

Transmitters  (4.2)  _  _  _ 

Receivers  (4.3)  _  _  _ 


Please  circle  the  number  which  best  expresses  how  much  you  would  agree  with 
each  of  the  following  statements:  1  *  disagree  strongly  and  5  ■  agree 
strongly. 

**************** 

II.  From  your  general  knowledge  of  and  experience  with  simulators,  do  you 
think  that  including  simulators  in  training: 

disagree  agree  don't 

strongly  strongly  know 

a.  is  a  good  idea  12345  ? 


a.  is  a  good  idea 

b.  can  be  more  effective  than  actual 
equipment 

c.  can  provide  equivalent  training  with 
actual  equipment 

d.  must  look  like  actual  equipment 

e.  can  provide  adequate  training  at  a 
cost  savings 

f.  allows  for  more  complex  training 
problems 

g.  is  more  reliable  than  actual  equipment 

h.  teaches  safety  training  better  than 
actual  equipment 

i.  provides  more  variety  of  training  than 
actual  equipment 


2  3  4  5 


2  3  4  5 


2  3  4  5 


2  3  4  5 


1  2  3  4  5 


2  3  4  5 


2  3  4  5 


3  4  5 


1V»VT( 


disagree 

strongly 


agree  don  t 
strongly  know 


j.  is  something  you  would  use  as  an 

integral  part  of  your  teaching  program 


1  2  3  4  5 


1 


Circle  the  number  which  best  shows  how  important  you  think  each  of  the 
following  factors  is  in  evaluating  the  utility  of  simulators  for  providing 
training.  1  =  unimportant  -  5  =  very  important 


g- 


capability  of  the  lessonware  to  meet 
a  broad  range  of  Navy  training 
objectives 


unimportant 
1  2  3 


ability  to  simulate  complexity  of  actual  1 
problems  encountered  in  the  field 


a  lower  cost  of  hardware  and  operating 
expenditures  compared  to  actual 
equipment 


a  high  degree  of  similarity  of  the 
simulated  equipment  to  the  actual 
equipment 


a  savings  in  the  amount  of  time 
required  for  training 


f.  the  degree  of  control  of  Navy  personnel 
over  the  design  of  the  equipment 


the  capability  of  Navy  personnel  to 
modify  existing  or  create  new  lessons 
for  the  training  simulator 


h.  mobility  of  the  equipment,  for 

versatility  of  use  in  the  classroom 


i.  reliability  of  performance  of  the 
equipment 


j.  ease  of  maintenance  of  the  equipment 


very 

important 


don' t 
know 


k.  ability  to  more  closely  monitor  student  12345 
performance  on  the  equipment 


unimportant  important  know 


l.  a  greater  variety  of  material  covered  12345  ? 

in  lessons  compared  to  actual 

equipment 

m.  ease  of  use  for  the  training  staff  12345  ? 

**************** 

IV.  Please  indicate  the  amount  of  experience  you  have  had  with  the  2D  EEMT 
subsystem  (check  all  appropriate  statements). 

a.  _  have  heard  about  it,  but  never  actually  used  it 

b.  _  have  seen  a  demonstration  of  it 

c.  _  have  had  training  on  how  to  use  it  in  the  classroom 

d.  _  have  had  limited  use  of  it,  as  a  reference  for  teaching 

e.  _  have  used  it  as  a  regular  part  of  my  teaching 

f.  _  have  been  involved  with  writing  lessons  for  use  on  it 

g.  _  have  been  involved  with  the  design  and  development  of  the 

unit 

h.  _  have  not  heard  about  it,  seen  it,  or  used  it  (if  checked, 

please  skip  the  next  section) 


From  your  experience  with  the  2D  EEMT  subsystem,  to  what  extent  would  you 
agree  with  the  following  statements  about  the  free-play  troubleshooting 
lessons: 


a.  the  lessons  are  well  designed  for 
laboratory  instruction 

b.  the  lessons  are  well  designed  for 
remediation  instruction 

c.  the  lessons  meet  course  objectives 


disagree 

agree 

don* 

strongly 

strongly 

know 

1  2 

3 

4  5 

7 

1  2 

3 

4  5 

7 

1  2 

3 

4  5 

? 

To  what  extent  would  you  agree  with  the  following  statements  about  the 
tutorial  lessons: 


disagree 

strongly 

agree 

strongly 

don' 

know 

the  lessons 
laboratory 

are  well  designed 
instruction 

for 

1 

2 

3 

4  5 

1 

the  lessons 
remediation 

are  well  designed 
instruction 

for 

1 

2 

3 

4  5 

? 

the  lessons 

meet  course  objectives 

1 

2 

3 

4  5 

I 


Overall,  from  your  experience  with  the  2D  EEMT  subsystem,  to  what  extent 
would  you  agree  with  the  following  statements: 


disagree 

strongly 


agree  don't 

strongly  know 


g.  it  provides  a  useful  supplement  to 
operational  equipment  training 

h.  the  authoring  software  is  easy  to  use 

i.  the  authoring  software  provides  an 
effective  method  of  developing  new 
lessons 

j.  the  record  keeping  features  of  the  2D 
simplify  the  task  of  individual  student 
management 

k.  on  a  class  basis,  EEMT  system 
management  does  not  add  significantly 
to  the  workload 

l.  student  data  produced  by  the  EEMT 
system  are  helpful  in  student 
management 

m.  student  data  produced  by  the  EEMT 
system  are  useful  in  assessing  student 
abilities 


***************** 


Please  indicate  the  amount  of  experience  you  have  had  with  the  3D  EEMT 
subsystem  (check  all  appropriate  statements). 

a.  _  have  heard  about  it,  but  never  actually  used  it 

b.  _  have  seen  a  demonstration  of  it 

c.  _  have  had  training  on  how  to  use  it  in  the  classroom 

d.  _  have  had  limited  use  of  it,  as  a  reference  for  teaching 

e.  _  have  used  it  as  a  regular  part  of  my  teaching 

f.  _  have  been  involved  with  writing  lessons  for  use  on  it 

g.  _  have  been  involved  with  the  design  and  development  of  the 

unit 

h.  _  have  not  heard  about  it,  seen  it,  or  used  it  (if  checked, 

please  skip  the  next  section) 


2  3  4  5 


2  3  4  5 


2  3  4  5 


2  3  4  5 


2  3  4  5 


2  3  4  5 


2  3  4  5 


From  your  experience  with  the  3D  EEMT  subsystem,  to  what  extent  would  you 
agree  with  the  following  statements: 


disagree 

strongly 


agree  don'i 

strongly  know 


a.  the  hardware  is  too  simple  for  it  to 
be  an  effective  training  instrument 


1  2  3  4  5 


? 


b.  the  lessons  are  well  designed  for 
instruction  purposes 


1  2  3  4  5 


c.  the  lessons  meet  course  objectives 

d.  the  simulator  is  a  better  training 
instrument  than  the  operational 
equipment 

e.  if  given  the  opportunity,  I  will  use 
the  EEMT  as  an  integral  part  of  my 
teaching  program 


1  2  3  4  5 

1  2  3  4  5 

1  2  3  4  5 


? 


What  changes  do  you  think  are  needed  to  make  the  3D  EEMT  a  better 
training  device? 


***************** 


Any  additional  comments  regarding  the  design  and  use  of  the  EEMT  system' 


Section  A -5 


STRUCTURED  INTERVIEW  QUESTIONS 


In  your  estimation,  how  is  the  2-D  EEMT  system  compatible  with  the  ET 

School  laboratory  environment?  How  is  it  incompatible? 

a.  Does  the  EEMT  system  make  appropriate  use  of  existing  technical  aids, 
guides,  and  manuals  (FOMMS)? 

b.  Does  the  system  require  special  operational  services,  such  as  greater 
than  normal  electrical  power,  air  conditioning,  lighting,  or  space? 

c.  How  is  the  EEMT  system  compatible  with  remedial  or  voluntary  study 
laboratory  environments?  How  is  it  incompatible? 

d.  How  is  the  EEMT  system  configuration  compatible  and/or  incompatible 
with  team  vs.  individual  training? 


How  is  training  for  new  operations  or  maintenance  personnel  currently 
conducted? 

a.  How  much  time  is  currently  allocated  for  training  new  operations  or 
maintenance  personnel  (formal  training  vs.  OJT)? 

b.  Can  training  for  EEMT  operations  personnel  be  adequately  delivered  in 
2  weeks  of  formal  training  plus  CUT? 

Can  training  for  EEMT  maintenance  personnel  be  adequately  delivered 
in  4  weeks  of  formal  training  plus  CUT? 

c.  Is  current  operations  or  maintenance  personnel  training  for  EEMT 
adequate?  What  could  be  added  to  it  to  improve  the  training? 

d.  If  you  were  in  attendance  at  the  EEMT  Maintenance  Training  Course 
(October  25-30,  1981),  do  you,  based  on  your  present  experience  with 
the  EEMT  system,  believe  that  it  was  a  valuable  course  which  provided 
adequate  training? 

What  improvements  would  you  suggest  for  other  courses  of  this  type? 


In  what  ways  does  the  EEMT  system  incorporate  basic  safety  principles 
appropriate  to  safe  operation  of  the  equipment? 

In  what  ways  does  the  EEMT  system  incorporate  unsafe  operating 
conditions? 


V. 


% 


l 


■s.' 


a.  In  what  ways  does  the  EEMT  system  train  to  basic  safety  principles? 

In  what  ways  does  the  EEMT  system  train  to  unsafe  operating 
principles? 

b.  How  does  the  EEMT  system  incorporation  of  basic  safety  or  unsafe 
operating  conditions  compare  to  the  operational  equipment? 

How  does  the  EEMT  system  training  of  basic  safety  or  unsafe  operating 
conditions  compare  to  the  operational  equipment? 


4.  How  well  is  the  EEMT  system  capable  of  training  to  appropriate  levels  of 
electronic  maintenance  skills? 

a.  How  well  does  the  EEMT  system  train  to  appropriate  vacuum  tube,  solid 
state  and/or  LSI  technology? 

b.  How  effective  is  the  presentation  of  material  to  the  students  using 
equipment  images? 

Are  these  images  an  advantage  or  disadvantage  to  the  training  of  the 
electronics  material? 

c.  How  effective  is  the  presentation  of  EEMT  system  material  to  the 
concepts  of: 

drill  and  practice? 

cognitive  or  logical  process? 

team  vs.  individual  training? 

tutorial  vs.  free  play  lessons? 

d.  In  what  ways  are  free  play  lessons  more  appropriate  than  tutorials  to 
the  objectives  of  Area  I  training? 

In  what  ways  are  free  play  lessons  less  appropriate  to  the  objectives  of 
Area  I  Radar  than  tutorials? 

e.  How  easily  can  the  EEMT  system  adapt  to  changing  technologies, 
objectives,  or  job  demands? 

f.  How  easily  can  EEMT  operations  personnel  modify  the  EEMT  system  to 
adapt  to  changing  technologies,  objectives,  or  job  demands? 


How  easy  is  it  to  understand  the  EEMT  system  technical  documentation 
(operations,  maintenance,  systems  programming)? 
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a.  How  easy  is  it  to  use  the  EEMT  system  technical  documentation? 

b.  How  complete  and  comprehensive  is  the  EEMT  technical 
documentation? 

c.  How  available  to  EEMT  system  operators  or  maintainers  is  the  EEMT 
technical  documentation? 

d.  What  process  is  involved  in  revising  or  modifying  the  EEMT  technical 
documentation? 

e.  How  frequently  is  it  necessary  to  revise  or  modify  the  EEMT 
documentation,  or  how  frequently  should  it  be  done? 
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Table  B-l 


Primary  Power  Performance  Test  Scores  as  a  Function  of  Student 
Ability  Levels  and  Training  Equipment  Availability  for  Historical  Data  Samples 


Training  Equipment  Availability 


Student  Ability 

* 

Operational 
Equip.  Only 

EEMT  Extra 
Study  Only 

EEMT  Lab  Plus 
Extra  Study 

Overall 

N 

24 

24 

24 

72 

Low 

X 

73.92 

82.17 

83.54 

79.88 

o 

20.72 

19.31 

13.08 

18.27 

N 

24 

24 

24 

72 

Middle 

X 

80.42 

88.73 

89.23 

86*  12 

a 

21.11 

18. 80 

10.18 

17.57 

N 

24 

24 

24 

72 

High 

X 

85.34 

92.75 

97.16 

91.75 

o 

15.82 

10.65 

5.89 

12.37 

N 

72 

72 

72 

216 

Overall 

X 

78.89 

87.88 

89.98 

85.92 

a 

19.66 

17.07 

11.48 

16.92 

♦Defined  by  ET  School  cumulative  grade  average  prior  to  AN/SPS-10 
Primary  Power. 


Training 

Mode 

Training 

Format 

EEMT 

Laboratory 

Training* 

EEMT 

Extra  Study/ 
Remediation 

EEMT 

Total** 

Operational 

Equipment 

High  EEMT 

Individual 

N 

27 

27 

27 

27 

Supplement 

X 

3.59 

1.04 

4.63 

3.78 

CT 

2.10 

1.43 

2.57 

1.12 

Pair 

N 

23 

22 

24 

24 

X 

4.26 

1.36 

5.50 

4.63 

a 

2.67 

1.84 

2.93 

1.79 

Overall 

N 

50 

49 

51 

51 

X 

3.90 

1.18 

5.04 

4.18 

a 

2.38 

1.62 

2.76 

1.52 

Low  EEMT 

Individual 

N 

28 

28 

28 

28 

Supplement 

X 

1.57 

1.00 

2.61 

6.29 

o 

1.32 

2.16 

2.63 

1.72 

Pair 

N 

27 

27 

27 

27 

X 

2.41 

1.30 

3.70 

6.78 

o 

1.55 

2.07 

2.84 

1.67 

Overall 

N 

55 

55 

55 

55 

X 

1.98 

1.15 

3.15 

6.53 

0 

1.48 

2.10 

2.77 

1.70 

Operational 

Individual 

N 

24 

Equipment 

X 

- 

- 

- 

7.46 

Only 

a 

- 

- 

- 

2.25 

Pair 

N 

27 

X 

- 

- 

- 

7.63 

0 

* 

- 

- 

2.10 

Overall 

N 

51 

X 

- 

- 

7.55 

_ 

• 

• 

2.15 

♦Training  also  included  the  "normal  operation"  and  "light  off  lessons  for  all 

students. 

♦♦While  the  EEMT  total  is  the  sum  of  laboratory  and  extra 
study/remediation,  slight  variations  can  result  from  missing  data. 


Table  B-4 


Supervisors  rated  technicians  on  a  five-point  scale  for  each  item  where  1  =  one 
of  the  worst,  2  =  below  average,  3  =  average,  4  =  above  average,  and  5  a  one  of  the  best. 


Table  B-5 


Instructor  Critique  Results 


Selected  Questions 

Mean  Rating 
of  Agreement* 

Number  of 
Respondents 

Standard 

Error 

From  your  general  knowledge  of  and  experience 
with  simulators,  do  you  think  that  including 
simulators  in  training: 

o  is  a  good  idea 

3.66 

47 

.18 

o  can  be  more  effective  than  actual  equipment 

1.72 

47 

.15 

o  can  provide  equivalent  training  with 
actual  equipment 

2.20 

46 

.17 

o  must  look  like  actual  equipment 

2.83 

42 

.23 

o  can  provide  adequate  training  at  a  cost  savings 

2.32 

37 

.17 

o  allows  for  more  complex  training  problems 

2.59 

46 

.21 

o  is  more  reliable  than  actual  equipment 

1.87 

46 

.14 

o  teaches  safety  training  better  than 
actual  equipment 

1.30 

47 

.08 

o  provides  more  variety  of  training  than 
actual  equipment 

2.43 

47 

.20 

o  is  something  you  would  use  as  an  integral 
part  of  your  teaching  program 

2.94 

46 

.18 

Table  B-5  (cont.) 


Selected  Questions 

Mean  Rating 
of  Importance* 

Number  of 
Respondents 

Standard 

Error 

How  important  do  you  think  each  of  the  following 
factors  is  in  evaluating  the  utility  of  simulators 
for  providing  training: 

•  capability  of  the  lessonware  to  meet  a  broad 
range  of  Navy  training  objectives 

4.26 

46 

.14 

•  ability  to  simulate  complexity  of  actual 
problems  encountered  in  the  field 

4.30 

46 

.15 

•  a  lower  cost  of  hardware  and  operating 
expenditures  compared  to  actual  equipment 

3.78 

45 

.18 

•  a  high  degree  of  similarity  of  the  simulated 
equipment  to  the  actual  equipment 

3.87 

46 

.20 

•  a  savings  in  the  amount  of  time  required 
for  training 

3.40 

45 

.19 

•  the  degree  of  control  of  Navy  personnel  over 
the  design  of  the  equipment 

3.56 

45 

.16 

•  the  capability  of  Navy  personnel  to  modify 
existing  or  create  new  lessons  for  the 
training  simulator 

4.49 

47 

.11 

•  mobility  of  the  equipment,  for  versatility 
of  use  in  the  classroom 

3.15 

40 

.21 

•  reliability  of  performance  of  the  equipment 

4.68 

47 

.11 

•  ease  of  maintenance  of  the  equipment 

4.63 

46 

.08 

•  ability  to  more  closely  monitor  student 
performance  on  the  equipment 

4.17 

47 

.14 

•  a  greater  variety  of  material  covered  in 
lessons  compared  to  actual  equipment 

3.84 

45 

.17 

•  ease  of  use  for  the  training  staff 

4.22 

46 

.15 

*scale:  1  =  unimportant,  5  =  very  important 

b-'; 


Table  B-5  (cont.^ 


Selected  Questions 

Mean  Rating 
of  Agreement* 

Number  of 
Respondents 

Standard 

Error 

From  your  experience  with  the  2D  EEMT  subsystem, 
to  what  extent  would  you  agree  with  the  following 
statements  about  the  free-play  troubleshooting 
lessons: 

•  the  lessons  are  well  designed  for 
laboratory  instruction 

2.74 

46 

.15 

•  the  lessons  are  well  designed  for 
remediation  instruction 

3.26 

46 

.16 

•  the  lessons  meet  course  objectives 

3.11 

45 

.16 

Overall,  from  your  experience  with  the  2D  EEMT 
subsystem,  to  what  extent  would  you  agree  with 
the  following  statements: 


it  provides  a  useful  supplement  to 
operational  equipment  training 

3.32 

47 

.20 

the  authoring  software  is  easy  to  use 

3.06 

35 

.22 

the  authoring  software  provides  an  effective 
method  of  developing  new  lessons 

2.39 

26 

.26 

the  record  keeping  features  of  the  2D  simplify 
the  task  of  individual  student  management 

2.78 

45 

.20 

on  a  class  basis,  EEMT  system  management 
does  not  add  significantly  to  the  workload 

2.42 

45 

.17 

student  data  produced  by  the  EEMT  system 
are  helpful  in  student  management 

2.42 

45 

.15 

•  student  data  produced  by  the  EEMT  system 
are  useful  in  assessing  student  abilities 


2.36 


45 


.17 


Table  B-6 


Student  Critique  Results 


Mean  Rating 

Selected  Questions  of  Agreement* 

Number  of 
Respondents 

From  your  experience  with  the 
operational  AN/SPS-10  radar 
equipment,  to  what  extent  would 
you  agree  with  the  following 
statements: 

•  There  was  enough  training  time 
on  the  equipment 

3.15 

108 

•  Training  time  on  the  equipment 
was  often  wasted 

1.65 

108 

•  A  variety  of  training  problems 
were  provided 

3.97 

106 

•  Equipment  breakdowns  interfered 
with  training 

1.54 

108 

•  Training  time  was  mostly  spent 
on  troubleshooting 

3.75 

109 

•  More  time  should  be  spent 
on  troubleshooting 

3.83 

105 

•  I  have  a  good  understanding  of 
how  to  maintain  this  equipment 

3.77 

107 

•  1  feel  comfortable  working 
on  this  equipment 

4.09 

108 

♦scale:  l=not  at  all,  5=very  much 


Table  B-6  (cont.) 


Mean  Rating 
of  Agreement* 


Number  of 
Respondents 


Standard 

Error 


B-10 


Table  B-7 


Frequent  Comments  from  Student  Critiques 
Concerning  2D  EEMT  Training 


Comment 


Reliability: 

•  general  malfunctions  and 
system  bugs 

•  touch  panel  drift 

•  system  lock-up  problem 

Fidelity: 

•  realistic  representation/ 
not  realistic  enough 

•  tells  student  if  a  relay 
is  energized,  closes 
interlocks,  hooks  up 
meters,  etc. 

•  set-up  and  use  of  test 
equipment 

Lesson  ware: 

•  simple/too  simple;  you  can 
easter-egg  problems 

•  variety  of  problems; 
component  rather  than 
wire  problems 

•  color/poor  picture  quality; 
cannot  read  reference 
designations 

•  contributes  to  bad  safety 
habits  (or  fails  to  support 
good  ones) 

System/Lesson  Operation: 

•  easy  to  use;  convenient 

•  comfortable 

•  faster  than  operational 
equipment;  able  to  solve 
more  problems 

•  too  slow 

•  hard  to  access  components; 
lots  of  moves  retired 

•  reduced  safety  risk 


Number  of  Respondents 
Like  Dislike 


-truer*-  -C” <rk  v; v. '  ■  - . 


Table  B-7  (cont.) 


Comment 


Number  of  Respondents 
Like  Dislike 


General: 

•  positive/ negative  2  4 

•  good  troubleshooting  practice  8  0 

•  availability  in  extra  study  3  1 

•  need  to  hear  blowers,  etc.;  0  3 

too  easy  to  forget  settings 

•  gave  good  system  overview;  6  0 

helped  with  system 

familiarity 


2D  Contract-Related  Data 


Basic  Contract  //N00123-80-R-032 7  (2D  only)  -  Duration:  ]une,  1980- 
Dec  ember,  1981 

Purpose:  This  contract  was  for  the  purpose  of  providing  20  2D  EEMT  units 


Total  Contract  Costs  were: 


Cost 

$  485,551 

Fee 

48,555 

CAS 

3,546 

Total 

$  537,652 

:ract  Line  Items  Included: 

20  EEMT 

$  402,229 

2  inst  terminals 

2,143 

Technical  support 
services 

9,233 

Manufacturer’s  training 
course 

6,092 

Technical  data 

114,409 

CAS  414  (CMF) 

Total 

$  537,652 

c.  Proposed  cost  of  additional  units  was: 


Cost 

$ 

15,720 

Fee 

1,572 

CMF 

_ _ 

53 

Total 

$ 

17,345 

EEMT  Lessonware  Development:  Contract  Modifications  POl,  PQ2  -Duration: 
October,  1980-April,  1982 

Purpose:  These  contract  modifications  were  necessary  for  the  development 
of  training  software  and  lessonware  for  2D  and  3D  EEMT  units. 

a.  This  task  included  development  of: 

o  AN/SPS-10  -  tutorial  free  play  lessonware 


s 


•  AN/SLQ  32  (ESM)  tutorial  lessonware 

•  Data  bases  for  the  AN/SPS-10,  AN/SLQ-32,  AN/WSC-3,  and  generic 
training  programs 


b.  Total  contract  costs  for  this  task  were: 


Total 


c.  Contract  line  items  were: 


$  710,524 
67,500 
5,963 

$  783,987 


Program  management  $  36,751 

Configuration  management  44,940 

Data  Collection  for  lessonware  186,  880 

Lessonware 

authoring/data  base  generation  250,554 

Modifications  to  software  to  drive 
3D  NPRDC  hardware  system  68,408 

Technical  publications  118,878 

Support  services  4,110 

CAS  $  5,963 


Total 


$  710,504 


3.  Convert  to  Videodisc  and  Touch  Panels:  Modification  P06  -  Duration:  Duly, 
1981 -December,  1981 

Purpose:  This  contract  modification  was  for  the  purpose  of  changing  the  2D 
EEMT  system  from  Bruning  microfiche/sonic  pens  to  videodisc  and  touch  panels. 
Costs  shown  include  the  hardware  (adjusted  against  cost  of  previous  hardware); 
software;  documentation  changes 


a.  Total  Contract  Cost: 


Total 


$  53,322 
532 

$  53,854 


4.  Cost  of  Government  Furnished  Equipment  -  Duration:  3une,  1977-December, 
1979,  (approximately) 

a.  The  University  of  Southern  California  received  approximately  $488,000  over 
2ft  years  for  training  system  software,  basic  GMTS  design/configuration,  and 
free  play  strategy  software. 


b.  Contract  costs  for  6  GMTS  units  included: 


$8-10,000  terac  per  each 
$5,000  graph  pen 

Total  for  basic  equipment 

$  48-6QK 

3  OK 

$  78-90K 

Modify  lessonware  and  GFE  software: 
1981-May,  1983. 

Modification  P07 

-  Duration:  September, 

Purpose:  This  contract  modification  was  required  to  modify  government 

furnished  software  for  training  system  and  free  play  strategy.  Because  of  the 
hardware  changes,  software  modifications  were  necessary,  including  all  data 
base  structures  and  the  reentry  of  all  lessonware. 

a.  Total  Contract  Costs  were: 

Cost 

Fee 

$  455,307 
44,607 

Total 

$  499,914 

b.  Line  items  included: 

(LI)  The  update  and  revision  of  government  furnished 

syllabus/curriculum  documentation  for  representative 

systems  lesson  specifications.  Revisions  were  made  to: 

9  SPS-10  lessons  and  18  GFE  SLQ-32  lessons 

Changes  required  deletion  of  5  original  AN/SPS-10 
lessons  and  the  creation  of  6  new  lessons  @  $53,255 

(L2)  The  redesign  of  government  furnished  lessonware 

specification  for  generic  systems.  Twenty-six  generic 

lessons  (87)4  hours)  were  modified  for  the  redesigned  3D 
EEMT. 

(This  is  a  3D  task)  199,146 

(L3)  Modification  of  lesson  specifications  for  test 

equipment  13,531 

(L6)  Expanded  AN/SPS-10  free  play  lessonware  from 
the  original  1 5  fault  problems  to  30  problems. 

The  15  new  lessons  included  7  power  supply  fault 
isolations  problems  and  8  AC  voltage  regulator 
isolation  problems.  Costs  for  this  lessonware 
development  were  $3,802.60  per  lesson. 


57,039 


Purchase  of  20  2D  units:  Modification  P09  -  Duration:  November,  1981  -May, 


1982 


Purpose:  This  contract  modification  was  to  allow  the  purchase  of  20 

additional  2D  units 


Total  contract  costs  were: 


u 


Total 


$  315,460  (20(3  17,345/unit— basic 

contract  price) 

31,440 
$  346,900 


Data  base  restructuring:  Modification  PI  5  -  Duration:  April,  1983  -  July, 
1983. 

Purpose:  This  contract  modification  was  to  allow  the  addition  of  a  step 
control  function  in  steps  1 6  and  1 7  of  the  new  data  base 


Total  Contract  Costs  were: 


Total 

8.  Refund:  Modification  PI  6 


$  69,601 

6,776 

$  76,377 


Purpose:  This  contract  modification  was  for  the  refund  of  $  130,000  for 
software  development  not  required  due  to  other  contract  modifications 

9.  Refund:  Modification  17 

Purpose:  This  contract  modification  was  for  the  refund  of  $50  in  fees  to  the 
government. 

i  0.  Spare  Parts  Contract 


Spare  parts  for  the  2D  EEMT  units  consist  of  components  purchased  and 
delivered  by  NPRDC  and  components  delivered  by  Cubic  Corporation.  The 
total  value  assigned  to  this  spares  package,  as  determined  from  the  proposed 
spares  contract  and  conversations  with  the  project  monitor,  is: 


Cubic  spares  package 
NPRDC  purchases 

Total 

20  2D  EEMT  units 
per  unit 


$  66,279 
36,959 

$  103,238 

t  20 

$  5,162 


Cost  Notes  for  Tables  8,  9,  10,  11,  12,  13,  14 


A  The  total  cost  of  the  station/console  includes  the  following  line  items  listed 
in  contract  modification  P06: 


Desk 

Casters 

Rear  Panels 

Enclosures 

Drawer 

Modification 


$  626.56 

22.36 
49.84 
400.00 
26.80 
25.00 


$  1,150.56 


B  The  total  cost  of  system  interfaces/controllers  includes  the  following  line 
items  listed  in  contract  modification  P06: 


A/N  Controller 

$ 

280.00 

Graphic  Controller 

375.00 

Electronic  Chassis 

50.00 

W/W  Board 

233.47 

ICS 

76.80 

Wire 

45.00 

Power  Strips 

28.40 

Touch  Panel  Interface 

125.00 

Cabling 

25.00 

Ribbon  Cables 

— 

75.00 

$  1 

,313.67 

C  These  cost  estimates  are  shown  in  the  Basic  Contract  as  a  "miscellaneous" 
line  item. 


D  Fabrication  for  the  first  copy  was  determined  by  using  the  cost  for  an 
additional  unit  based  on  contract  price  and  subtracting  the  cost  of  freight  and 
et^iipment  costs  (also  based  on  contract  figures)  and  adding  the  fee  for  one 
terminal.  This  resulted  in  the  following  calculation: 


Additional  Unit 


Basic  Contract  price 

$ 

1  7,  345. 00 

Less  freight 

5" 

-  112.50 

Cost  per  unit 

17,232.50 

Printer 

974.00 

5" 

18,206.50 

Less  Equipment  Cost 

- 

11,534.00 

r 

6,672.50 

Fee  for  Terminal 

— 

105.00 

Total  fabrication  cost 

$ 

6,777.50 

Additional  copy  costs  do  not  reflect  the  terminal  fee. 


D-2 


E  The  prototype  development  costs  were  determined  from  Basic  Contract 
costs.  Relevant  modifications  (P06)  were  added  and  the  costs  of 
sof tware/ lesson w -.re  were  subtracted.  Seventy  percent  of  technical  data 
costs  (as  determined  from  Item  5,  Basic  Contract)  and  technical  support  costs 
were  added.  The  additional  unit  cost  times  20  units  was  subtracted.  Finally, 
government  furnished  equipment  costs  were  added.  The  resulting  calculation 
was  as  follows: 

Total  Basic  Contract 
Videodisc  format  modifications 
(P06) 

Software/lessonware 
70%  Engineering  Technical  Data 
Technical  Support 
20  Additional  2D  EEMT  Units 
GFE  (hardware) 

$  195,237  Development  costs 

F  Prototype  user  documentation  reflected  the  remaining  30  percent  of  the 
technical  data  cost  from  contract  documents  (see  Cost  Note  E).  Five  percent 
was  considered  adequate  for  first  copy  costs. 

G  The  cost  of  shipping  the  2D  EEMT  to  its  destination  is  not  included  in  these 
cost  estimates.  Note  that  the  total  hardware  cost  of  additional  units 
($17,232)  plus  a  shipping  charge  ($113)  equals  the  $17,345  paid  by  the 
government  for  each  additional  2D  EEMT  unit. 

H  Software  development  costs  were  calculated  using  GFE  software  figures  and 
adding  contract  modification  amounts  from  EEMT  software  and 
documentation  enhancement  modification  (P07)  and  data  base  structure 
modifications  (PI 5).  Task  S5  of  P07  was  not  included  as  it  addressed  the  3D 
EEMT  software  only.  This  results  in  the  following  calculation: 

GFE  software 

Strategy  specification  (P07,  SI) 

Operating  system  (P07,  S3) 

Software  consolidation  (P07,  S4) 

Data  base  structure  modifications 
(PI  5) 

$  313,836 

I  Next  copy  and  additional  copy  costs  were  assumed  to  be  $50  for  diskettes. 


$  160,000 
19, 152 
19,022 
39,285 

76,377 


3  Prototype  copy  structured  lessonware  development  costs  were  derived  from 
contract  modification  EEMT  lessonware  development  (P02)  less  costs  for  3D 
lessonware  and  documentation  of  generic  lessonware.  In  addition,  the  cost  of 
free  play  lessonware  for  Primary  Power  radar  was  subtracted  out.  The 
resulting  amount  was  reduced  by  80  percent  to  reflect  Primary  Power 
tutorial  lessonware  development  only.  GFE  lessonware  costs  were  added. 
Primary  Power  free  play  lessonware  costs  were  then  added.  Videodisc 
production  costs  were  determined  from  contract  modifications  (P06).  Money 
refunded  for  3D  lessonware  not  developed  was  subtracted.  Lessonware 
modifications  from  contract  modification  P07  were  added.  The  resulting 
calculation  was: 


GFE  Lessonware  $  160,000 

Free  play  Lessonware  114,078 

20%  Structured  Lessonware  96,525 

Videodisc  Production  17,222 

Miscellaneous  620 

Lessonware/software  30, 420 

modifications  P07 


$  418,865 

First  copy  development  costs  reflect  lesson  development  for  30  free  play  lessons, 
videodisc  production,  development  and  mastering. 


Lesson  development 

$  114,078 

30  free  play  lessons  at  $3,803  per 

lesson  (Source:  Contract  Modification 
P07,  L6) 

Videodisc  development 

$ 

1,000 

Photography 

750 

SME  support 

11,480 

Videodisc  support 

— 

335 

Camera,  film,  processing 

$ 

13,565 

(For  three  videodiscs) 

(Source:  Contract  Modification  P06) 

$ 

4,522 

Per  videodisc  production 

$ 

4,700 

Premastering  (Source:  Army  Signal 
Corps  Military  Interdepartmental 
Purchase  Request  0N6822182  MP20098) 

1,700 

Master  (Source:  Project  Monitor 
Communication) 

$ 

10,922 

Total  videodisc  production 

Videodisc  production  cost  differences  between  prototype  and  next  copies  reflect 
multiple  disk  production  and  demo  disk  production  costs  for  the  prototype  not 
reflected  in  the  next  copy. 


Lesson  development 

and  videodisc  development 

for  first  copy:  $  114,078 


9 

cl 


■ 


Lessonware  acquisition  costs  for  a  2D  EEMT  first  copy  and  an  additional  copy 
reflect  purchase  of  printer  paper,  floppy  disks  (20  (3  $7),  and  a  single  copy  of 
the  videodisc.  The  assumption  is  made  that  the  additional  copy  videodisc  is 
obtained  at  the  time  that  the  next  copy  unit  is  obtained,  avoiding  the 
remastering  costs  of  the  videodisc. 


Printer  paper 
Floppy  discs 
Videodisc 


$  480 
140 
_ 17 

$  637 


L  The  spares  package  amount  was  determined  from  contract  specifications  and 
conversations  with  the  NPRDC  Contract  Monitor  for  parts  purchased  and 
allocated  to  a  per  unit  base  (see  Appendix  C). 

M  Facilities  costs  are  based  on  T.  Eggemeier,  Life  Cycle  Cost  Estimation  of 
Simulated  vs.  Actual  Equipment  Maintenance  Training  for  the  F-16  Avionics 


Intermediate  Shop  (AFHRL  Advanced  Systems  Division,  Wright-Patterson 
AFB,  OH:  March  1979).  The  replacement  cost  of  space  in  that  study  was 
estimated  to  be  $16.27  per  square  foot  (sq.  ft.).  Adjusted  for  inflation  from 
1977  (original  date  of  estimate)  results  in  a  1981  replacement  cost  of  space 
of  $30  per  square  foot.  Space  allocated  for  the  eight  2D  EEMT  units  used  in 
the  Primary  Power  training  laboratory  was  200  sq.  ft.  For  the  10  operational 
equipments,  400  sq.  ft.  were  used.  All  costs  are  per  unit  costs.* 

N  Acquisition  management  was  included  in  development  costs  shown  in  Table  8. 

O  Investment  costs  for  the  2D  EEMT  reflect  the  spares  package  shown  in  Table 
8.  The  operating  year  sustaining  investment  costs  for  this  trainer  were 
assumed  to  be  5  percent  of  the  cost  of  the  equipment.  Actual  sustaining 
costs  for  the  2D  EEMT  were  complicated  by  the  establishment  of  commercial 
repair  contracts.  For  the  operational  equipment,  Navy  maintenance 
documents  available  for  the  TCT  were  considered  to  underestimate  the 
actual  costs  of  repair  over  a  life-cycle  period.  Actual  parts  costs  for  the 
period  of  January  -December  1983  were  determined  to  be  $191,  which  did  not 
account  for  major  system  components  which  could  fail  once  per  every  5  year 
cycle.  Therefore,  the  5  percent  of  equipment  cost  amount  shown  anticipates 
major  component  breakdown  spread  over  the  life-cycle.  All  costs  are  per 
unit  costs. 

P  Maintenance  costs  were  derived  from  personnel  allocations  to  2D  EEMT 
maintenance  with  pay  scales  from  the  Navy  Composite  Standard  Military 
Rate  Table  deflated  to  1981  rates  according  to  the  Consumer  Price  Index 
(CPI).  This  maintenance  level  assumes  one  ET3  and  two  ET2  level  personnel 
full-time.  It  also  includes  overhead  as  follows:  one  chief  petty  officer  at  10 
percent  and  one  petty  officer  first  class  at  25  percent,  based  on  staffing 
responsibility  for  30  and  12  persons,  respectively.  This  total  was  then 
allocated  over  the  40  2D  EEMT  units  over  which  the  maintenance  staff  had 
responsibility.  All  costs  are  per  unit  costs. 


m 


Q  Operating  cost  estimates  were  determined  from  conversations  with  the  base 
facilities  administrator.  Utilities  costs  were  estimated  to  be  $7  per  square 
foot  per  year.  The  2D  EEMT  laboratory  housing  S  2D  EEMT  units  was 
estimated  to  be  200  square  feet.  All  costs  are  in  per  unit  costs. 

R  Operating  year  FOMM  and  lessonware  materials  costs  were  determined  from 
the  annual  printing  costs  charged  to  the  AN/SPS-10  Radar  training  area.  No 
reduction  was  made  for  distribution  of  costs  over  all  of  the  AN/SPS-10  Radar 
areas,  since  the  full  set  of  manuals  and  workbooks  would  be  required  for  any 
one  section  of  AN/SPS-10  Radar  training.  Annual  printing  charges  to 
AN/SPS-10  Radar  areas  is  estimated  at  $63,250. 

S  Updating  of  laboratory  exercises/materials  was  estimated  at  1  week  per  year 
for  an  instructor  (GS-9,  Step  5  or  ET  3).  Operational  equipment  lessons  are 
altered  very  infrequently.  Existing  problems  cover  all  primary  areas  of  the 
equipment  and  can  be  randomly  assigned.  In  addition,  laboratory  instructors 
can  insert  equipment  faults  by  hand  as  needed.  For  the  2D  EEMT,  the 
existing  30  free  play  lessons  are  considered  a  sufficient  pool  for  the  life- 
cycle  of  the  equipment.  TCT  results  (Table  B-2)  show  fewer  than  five 
problems  worked  per  student.  Therefore,  allocated  time  is  for  debugging  and 
minor  changes.  The  cost  calculation  was  as  follows: 

$25,956  (GS-9,  Step  5  annual  salary,  in  1981  dollars) 

(Source:  Mark,  L.3.,  Kribs,  H.D.,  Schuler,  J.W.,  and  Brown, 
J.E.  A  life  cycle  cost  model  for  an  electronic  maintenance 
trainer  (NPRDC  Special  Report  82-33).  p.  A-13. 

$  10,543  (ET-3  annual  salary  in  1981  dollars)  (Source:  Navy  Composite 
Rate  Table,  1982) 

$  1 8, 250  average  of  two  salaries 

.  i-J2 

$  350  per  week 

T  Laboratory  instructor  costs  were  assumed  to  be  an  average  for  a  GS-9, 
Step  5/  ET3  pay  scale.  One  full-time  instructor  was  assumed.  See  Cost  Note 
R. 

U  Overhead  for  instructor  personnel  was  determined  to  be  at  55  percent. 
Source:  Mark,  L.3.,  Kribs,  H.D.,  Schuler,  3.W.,  and  Brown,  J.E.  A  life  cycle 
cost  model  for  an  electronic  maintenance  trainer  (NPRDC  Special  Report  82- 
33).  p.  A-13. 

V  The  calculation  for  determining  student  wages  for  ET  "A"  School  was  as 
follows: 


$23.85/day  x  4  training  days/week  x  66  students/week  x  50  weeks/year  = 
$314,820 


W  Miscellaneous  student  support  costs  were  estimated  to  be  $1 10/week  per 
student.  This  estimate  was  obtained  from  ET  "A"  School  personnel.  The 
calculation  is: 

$1 10/week/student  x  66  students/week  x  50  weeks/year  =  $363,000/year 

X  The  cost  of  an  operational  equipment  AN/SPS-lOwas  derived  from  estimates 
of  purchase  price  in  1979  ($37,500)  adjusted  to  19S1  dollars  based  on  the 
producer  price  index  of  the  US  GPO  Economic  Indicators. 

Y  Maintenance  costs  were  determined  from  RCA  personnel  allocations  and 
salary  schedules  and  apportioned  for  the  Primary  Power  laboratory 
AN/SP5- 10  units.  All  costs  shown  are  per  unit  costs. 

Z  Operating  costs  were  determined  from  conversations  with  the  base  facilities 
administrator.  Utilities  costs  were  estimated  to  be  $7  per  square  foot  per 
year.  The  operational  equipment  laboratory  was  determined  to  be  400  square 
feet  in  area.  All  costs  shown  are  per  unit  costs. 

AA  2D  EEMT  hardware  costs  include  two  next  copies  and  eight  additional  copies, 
based  on  the  assumption  that  the  next  copy  instructor  terminal  (printer)  can 
support  four  additional  copies.  All  other  equipment  costs  are  life-cycle  costs 
(Tables  9,  10,  11)  for  10  units. 

BB  AN/SPS-67  equipment  costs  were  derived  from:  Mark,  L.3.,  Kribs,  H.D., 
Schuler,  J.W.,  and  Brown,  J.E.  A  life  cycle  cost  model  for  an  electronic 
maintenance  trainer  (NPRDC  Special  Report  82-33).  p.  A-5. 

CC  AN/SPS-67  and  AN/SPS-10  curriculum  development  costs  were  derived  from: 
Mark,  L.J.,  Kribs,  H.D.,  Schuler,  3.W.,  and  Brown,  J.E.  A  life  cycle  cost 
model  for  an  electronic  maintenance  trainer  (NPRDC  Special  Report  82-33). 
p.  A-13  for  curriculum  development  costs.  Curriculum  maintenance  costs  for 
the  AN/SPS-67  were  assumed  to  be  comparable  to  estimates  for  the 
AN/SPS-10. 

DD  2D  EEMT  Lessonware 

15  free  play  lessons 
(1 14,078  r  2)  $ 

FOMMS/technical 
documentation 
($63,250  x  2  to 
represent  redevelopment 
of  all  documentation)  __ 

$  183,539 

AN/SPS-10  iessonware 
development 

(42,190  f  2)  $  21,095 

FOMMs/technica] 

documentation  126, 500 

$  147,595 


57,039 


126,500 


EE  One  next  copy  and  four  additional  copies  are  assumed.  Rf.  TabJe  8. 


FF  Structured  2D  EEMT  iessonware  development  costs  were  determined  from 
the  total  software/lessonware  development  contract  costs  (P02),  less  3D 
EEMT  software  modifications  costs  and  documentation  of  generic  Iessonware 
costs.  The  resulting  amount  was  reduced  by  free  play  Iessonware  costs  and 
further  reduced  by  80  percent  to  reflect  Primary  Power  structured 
Iessonware  costs.  The  calculation  is: 


Total  P02  cost 

$  783,987 

3D  software  modifications 

-  68,408 

Generic  Iessonware 
documentation 

-118,878 

Primary  Power  free  play 
Iessonware 

-114,078 

$  482,623 

Reduced  by  80%  to  show 
Primary  Power  structured 
Iessonware  only 

-386,098 

$  96,523 
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Air  Force  Human  Resources  Laboratory,  Brooks  Air  Force  Base  (2  copies) 
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